CVE-2011-3336: 
PHP vulnerability analysis and mitigation

CVE-2011-3336 is a vulnerability in the BSD implementation of libc's regcomp function that leads to denial of service through stack exhaustion. The vulnerability was discovered in 2011 and affects various BSD-based systems including NetBSD 5.1, OpenBSD 5.0, FreeBSD 8.2, and MacOSX (NIST NVD, CX Security).

The vulnerability exists in the regcomp() function of BSD libc implementation where recursion and poor memory management can lead to unexpected application termination. The issue specifically occurs in the regular expression compilation process where certain patterns can trigger uncontrolled recursion and memory exhaustion. The vulnerability has a CVSS base score of 7.8/10, with an impact subscore of 6.9/10 and exploitability subscore of 10/10 (CX Security).

When exploited, this vulnerability can cause denial of service through stack exhaustion or memory exhaustion. In applications using regcomp() to process untrusted input, specially crafted regular expressions can cause the application to crash or consume excessive system resources (Full Disclosure).

The vulnerability can be triggered using specially crafted regular expressions that cause excessive recursion or memory consumption. For example, patterns like '(.?)((((.*){1,100}){1,100}){1,100}){1,100}' can cause the system to run out of swap space. The vulnerability has been demonstrated to affect various applications including ProFTPD and web browsers (CX Security).

The recommended fix includes implementing memory usage limits (suggested at 128MB for one regcomp(3) call) and recursion limits (set to 256 levels). NetBSD has implemented these fixes in their libc implementation. For systems without patches, limiting the size of input passed to the regular expression engine or avoiding quantification nesting in expressions can help mitigate the issue (CX Security).

There were varying responses from different vendors regarding this vulnerability. While NetBSD worked to fix the flaws, Red Hat initially decided not to address similar issues in GNU libc, stating they did not consider it a security issue for client applications using regcomp() on untrusted input. This created some debate in the security community about the responsibility for securing regex implementations (CX Security).