GHSA-wjmf-p669-5m5p
Protego has exponential backtracking ReDoS in robots.txt URL wildcard matching
Quick fix
GHSA-wjmf-p669-5m5p — protego: upgrade to the fixed version with the command below.
pip install --upgrade 'protego>=0.6.2'Details
### Problem description
Protego constructs regular expressions to match URLs against `robots.txt` `Allow:` and `Disallow:` directives, see `protego._urlpattern._URLPattern._prepare_pattern_for_regex()`. Every `*` in the directive value is translated into a lazy `.*?` regex piece, thus a specially crafted directive value with many asterisks may produce a regex that freezes the parser due to exponential backtracking.
### Impact
Parsing a specially crafted `robots.txt` with `protego.Protego.parse()` and then trying to match an URL with `protego.Protego.can_fetch()` results in the latter call not returning for a period dependent on the length of the URL.
### Proof of concept
```python from protego import Protego
robotstxt = f""" User-agent: * Disallow: /{"*1" * 12}*Z """ rp = Protego.parse(robotstxt) url = "/" + "1" * 60 rp.can_fetch(url, "mybot") # freezes ```
Are you affected?
Enter the version of the package you're using.