CVE-2026-55520: 
Python vulnerability analysis and mitigation

CVE-2026-55520 is a Regular Expression Denial of Service (ReDoS) vulnerability in Protego, a Python library used by Scrapy for parsing and enforcing robots.txt rules. The vulnerability affects all versions up to and including 0.6.1, with version 0.6.2 containing the fix. It was disclosed on June 25, 2026 via a GitHub Security Advisory. The vulnerability carries a CVSS v4 base score of 7.1 (High) (GitHub Advisory).

The root cause lies in how Protego constructs regular expressions to match URLs against Allow: and Disallow: directives in robots.txt files (classified as CWE-400: Uncontrolled Resource Consumption and CWE-1333: Inefficient Regular Expression Complexity). Specifically, the protego._urlpattern._URLPattern._prepare_pattern_for_regex() method translates each * wildcard character in a directive into a lazy .*? regex piece. A specially crafted robots.txt containing a directive with many asterisks (e.g., Disallow: /*1*1*1...*Z) produces a regex susceptible to exponential backtracking when matched against a sufficiently long URL. Exploitation requires no authentication or special privileges — only the ability to serve a malicious robots.txt to a Protego-based crawler (GitHub Advisory).

Successful exploitation causes the protego.Protego.can_fetch() call to hang indefinitely (or for an extended period proportional to URL length), effectively freezing the parsing process and denying service to the affected crawler or application. The impact is limited to availability — there is no confidentiality or integrity impact. Applications that automatically fetch and parse arbitrary robots.txt files from external sources are most at risk, as an attacker-controlled web server can serve a malicious robots.txt to trigger the denial of service (GitHub Advisory).

A public proof-of-concept is included in the GitHub Security Advisory, demonstrating that as few as 12 repeated *1 patterns followed by *Z in a Disallow: directive, combined with a 60-character URL, is sufficient to freeze the parser. The vulnerability requires passive user interaction (a crawler must fetch the malicious robots.txt), but no privileges are required. Nessus plugin 323327 has been released to detect this vulnerability. There is no current evidence of in-the-wild exploitation, threat actor attribution, or CISA KEV catalog listing (GitHub Advisory, Tenable).

1. Set up a malicious web server: Host a robots.txt file at the root of a domain or IP address that a Protego-based crawler (e.g., Scrapy) is expected to crawl.
2. Craft the malicious robots.txt: Create a robots.txt with a directive containing many asterisk wildcards, such as:

User-agent: *
Disallow: /*1*1*1*1*1*1*1*1*1*1*1*1*Z

3. Trigger the crawler: Ensure the target crawler fetches a URL from the attacker-controlled domain, causing it to retrieve and parse the malicious robots.txt via protego.Protego.parse().
4. Induce the freeze: When the crawler calls protego.Protego.can_fetch(url, user_agent) with a sufficiently long URL (e.g., 60+ characters), the exponential backtracking in the generated regex causes the call to hang indefinitely, denying service to the crawler process (GitHub Advisory).

• Process: Crawler worker processes (e.g., Scrapy spiders) hanging or becoming unresponsive, with high CPU utilization sustained over time on a single thread.
• Logs: Crawler logs showing a robots.txt fetch from an external domain followed by no further activity or timeout errors; absence of expected crawl progress after robots.txt parsing.
• Network: Outbound HTTP/HTTPS requests to retrieve robots.txt from unexpected or newly encountered domains, particularly those with unusual Disallow: patterns containing many * characters.
• File System: If robots.txt files are cached locally, presence of files with Disallow: directives containing repeated wildcard patterns (e.g., /*1*1*1*1*...*Z) (GitHub Advisory).

Upgrade Protego to version 0.6.2 or later, which contains the fix for this vulnerability. Users of Scrapy or any other application depending on Protego should update the package via pip: pip install --upgrade protego. As a temporary workaround where upgrading is not immediately possible, restrict crawlers to trusted domains or implement a pre-processing step to reject robots.txt files containing directives with an excessive number of * wildcards before passing them to Protego (GitHub Advisory).

The vulnerability was reported and published by AdrianAtZyte (a Zyte/Scrapy maintainer) via the GitHub Security Advisory program on June 25, 2026. The fix was released promptly in Protego 0.6.2. Tenable added Nessus detection (plugin 323327) shortly after disclosure, indicating recognition within the security tooling community (GitHub Advisory, Tenable).