Binary package “python3-protego” in ubuntu oracular

Pure-Python robots.txt parser with support for modern conventions

 Protego parses directives set in robots.txt, including non-standard ones such
 as Request-Rate and Host. It also handles some typos in the directive names.