← Tool directory
TOOL PROFILE

Spatie Robots Txt

Check crawl rules for specified paths and user agents from PHP.

Open source

Community support

Upvotes come from SearchEngineAnswer members. GitHub stars and reader reviews are counted separately.

Claim this tool if you maintain it.

Best for
Crawling
Setup required
A Composer project with a PHP version supported by the package.
Costs to check
Budget for development and tests. Local fixture checks require no hosted service.
License in catalog
MIT

GitHub repository

GitHub stars
258
Forks
44
Last push
2026-11-06
Primary language
PHP

GitHub snapshot from . Stars count GitHub bookmarks. Reader ratings appear separately below.

How to install Spatie Robots Txt

  1. Check the repository's PHP requirements and run composer require spatie/robots-txt in a test project.
  2. Load the Composer autoloader and prepare a short local robots.txt fixture.

First tutorial for Spatie Robots Txt

  1. Instantiate the documented RobotsTxt parser and query allows for a blocked path and an allowed sibling.
  2. Check the values and inspect whyDisallows to identify which rule matched the blocked case.

Official installation and usage documentation ↗

Where Spatie Robots Txt fits

Use this to add robots.txt rule tests to a PHP deployment or audit workflow.

When to choose another tool
Use a browser or command-line HTTP client when you first need to check which robots.txt the server actually serves.
First evaluation
Build a local rules fixture and check a blocked path and an allowed control.
How to interpret the result
A crawl allowance is separate from indexability. Confirm search-engine-specific behavior where it matters.

Reviews and discussion

Reader reviews

Be the first to share your experience with this tool.

Discussion

Ask a question or share a setup tip.

Write a review or join the discussion

Reviews should describe your own experience. All submissions are moderated.

Flag an outdated detail

Include the field that needs updating and a source link. Copy your draft, then use the corrections policy to contact the editorial team.