← Tool directory
TOOL PROFILE

Google robots.txt parser

Test how a robots.txt file matches URL paths and crawler user agents.

Open source

Community support

Upvotes come from SearchEngineAnswer members. GitHub stars and reader reviews are counted separately.

Claim this tool if you maintain it.

Best for
Crawling
Setup required
C++ build tools or a compiled parser for your environment.
Costs to check
Allow time to build the library and maintain representative rule fixtures.
License in catalog
Apache-2.0

GitHub repository

GitHub stars
3,479
Forks
259
Last push
2026-04-01
Primary language
C++

GitHub snapshot from . Stars count GitHub bookmarks. Reader ratings appear separately below.

How to install Google robots.txt parser

  1. Clone the repository and install the build tools for the Bazel or CMake workflow.
  2. Follow the repository’s build instructions. Run its included tests before adding your own rule fixture.

First tutorial for Google robots.txt parser

  1. Create a robots.txt file with a broad disallow and a more specific allow. Test one URL in each branch for the intended user agent.
  2. Compare the parser result with your expected rules, then fetch the production robots.txt file separately to verify what is actually served.

Official installation and usage documentation ↗

Where Google robots.txt parser fits

Use the parser when you need repeatable tests of robots.txt rule matching before shipping a configuration change.

When to choose another tool
Use a live fetch when the question is which robots.txt file the site actually serves. Pair the fetch with the parser to check both delivery and rule behavior.
First evaluation
Create a file with a broad disallow and a more specific allow. Test a URL in each branch for the user agent you care about. Record the expected result and repeat the test against the production file.
How to interpret the result
A matching decision describes the supplied file and test inputs. HTTP delivery, crawler access, and indexing still need their own checks.

Reviews and discussion

Reader reviews

Be the first to share your experience with this tool.

Discussion

Ask a question or share a setup tip.

Write a review or join the discussion

Reviews should describe your own experience. All submissions are moderated.

Flag an outdated detail

Include the field that needs updating and a source link. Copy your draft, then use the corrections policy to contact the editorial team.