CVE-2026-55520: CWE-400: Uncontrolled Resource Consumption in scrapy protego

The scrapy protego library constructs regular expressions to match URLs against robots.txt Allow and Disallow directives by translating each asterisk (*) into a lazy '.*?' regex fragment. A specially crafted directive with many asterisks can produce a regex pattern that triggers exponential backtracking, causing the parser to freeze during URL matching. This uncontrolled resource consumption vulnerability (CWE-400) affects versions of protego prior to 0.6.2. The issue manifests when Protego.parse() processes a malicious robots.txt and subsequent calls to Protego.can_fetch() hang for a duration dependent on the URL length.

An attacker can cause a denial of service by supplying a malicious robots.txt file with crafted directives containing many asterisks. This leads to Protego's regex engine freezing during URL matching, resulting in application hang or resource exhaustion. There is no indication of privilege escalation or data compromise, but availability is impacted.

Patch status is not yet confirmed — check the vendor advisory for current remediation guidance. Users should monitor the official scrapy/protego project for updates addressing this regex exponential backtracking issue. Until a fix is available, avoid parsing untrusted robots.txt files with many asterisks in directives.