PYSEC-2026-3906
Protego has exponential backtracking ReDoS in robots.txt URL wildcard matching
Quick fix
PYSEC-2026-3906 — protego: upgrade to the fixed version with the command below.
pip install --upgrade 'protego>=0.6.2'Details
### Problem description
Protego constructs regular expressions to match URLs against `robots.txt` `Allow:` and `Disallow:` directives, see `protego._urlpattern._URLPattern._prepare_pattern_for_regex()`. Every `*` in the directive value is translated into a lazy `.*?` regex piece, thus a specially crafted directive value with many asterisks may produce a regex that freezes the parser due to exponential backtracking.
### Impact
Parsing a specially crafted `robots.txt` with `protego.Protego.parse()` and then trying to match an URL with `protego.Protego.can_fetch()` results in the latter call not returning for a period dependent on the length of the URL.
### Proof of concept
```python from protego import Protego
robotstxt = f""" User-agent: * Disallow: /{"*1" * 12}*Z """ rp = Protego.parse(robotstxt) url = "/" + "1" * 60 rp.can_fetch(url, "mybot") # freezes ```
Are you affected?
Enter the version of the package you're using.
Affected packages
References
- https://github.com/scrapy/protego/security/advisories/GHSA-wjmf-p669-5m5p[WEB]
- https://github.com/scrapy/protego/commit/785940181659bf440ba82f1da148fade5087e858[WEB]
- https://github.com/scrapy/protego[PACKAGE]
- https://github.com/scrapy/protego/releases/tag/0.6.2[WEB]
- https://pypi.org/project/protego[PACKAGE]
- https://github.com/advisories/GHSA-wjmf-p669-5m5p[ADVISORY]
- https://nvd.nist.gov/vuln/detail/CVE-2026-55520[ADVISORY]