← Tool directory
TOOL PROFILE

Protego

Test how a robots.txt file applies to a specified user agent and URL.

Open source

Community support

Upvotes come from SearchEngineAnswer members. GitHub stars and reader reviews are counted separately.

Claim this tool if you maintain it.

Best for
Crawling
Setup required
Python and a copy of the robots.txt text you want to evaluate.
Costs to check
The library is open source and runs locally against the rules you provide.
License in catalog
BSD-3-Clause

Project screenshot

Local run with sample data: Robots.txt access decisions
Source documentation ↗

GitHub repository

GitHub stars
97
Forks
39
Last push
2026-11-01
Primary language
DIGITAL Command Language

GitHub snapshot from . Stars count GitHub bookmarks. Reader ratings appear separately below.

How to install Protego

  1. Create and activate a Python virtual environment.
    python -m pip install protego
  2. Install Protego and import the Protego class.

First tutorial for Protego

  1. Pass the fixture text to Protego.parse to create a parser.
  2. Call can_fetch with each staging URL and the same crawler user agent.
  3. Compare the returned decisions with your expected allow and block cases.

Official installation and usage documentation ↗

Where Protego fits

Use it to regression-test robots.txt rules before releasing a revised configuration.

When to choose another tool
Use verified server logs when you need to establish what a particular crawler actually requested.
First evaluation
Create a fixture with a blocked path and a more specific allowed path, then evaluate both for the same user agent.
How to interpret the result
Record the parser decision alongside the exact fixture and user agent. Check the live response after deployment.

Reviews and discussion

Reader reviews

Be the first to share your experience with this tool.

Discussion

Ask a question or share a setup tip.

Write a review or join the discussion

Reviews should describe your own experience. All submissions are moderated.

Flag an outdated detail

Include the field that needs updating and a source link. Copy your draft, then use the corrections policy to contact the editorial team.