Spatie Robots Txt
Check crawl rules for specified paths and user agents from PHP.
Community support
Upvotes come from SearchEngineAnswer members. GitHub stars and reader reviews are counted separately.
Claim this tool if you maintain it.
- Best for
- Crawling
- Setup required
- A Composer project with a PHP version supported by the package.
- Costs to check
- Budget for development and tests. Local fixture checks require no hosted service.
- License in catalog
- MIT
GitHub repository
- GitHub stars
- 258
- Forks
- 44
- Last push
- 2026-11-06
- Primary language
- PHP
GitHub snapshot from . Stars count GitHub bookmarks. Reader ratings appear separately below.
How to install Spatie Robots Txt
- Check the repository's PHP requirements and run composer require spatie/robots-txt in a test project.
- Load the Composer autoloader and prepare a short local robots.txt fixture.
First tutorial for Spatie Robots Txt
- Instantiate the documented RobotsTxt parser and query allows for a blocked path and an allowed sibling.
- Check the values and inspect whyDisallows to identify which rule matched the blocked case.
Where Spatie Robots Txt fits
Use this to add robots.txt rule tests to a PHP deployment or audit workflow.
- When to choose another tool
- Use a browser or command-line HTTP client when you first need to check which robots.txt the server actually serves.
- First evaluation
- Build a local rules fixture and check a blocked path and an allowed control.
- How to interpret the result
- A crawl allowance is separate from indexability. Confirm search-engine-specific behavior where it matters.
Reviews and discussion
Reader reviews
Be the first to share your experience with this tool.
Discussion
Ask a question or share a setup tip.
Flag an outdated detail
Include the field that needs updating and a source link. Copy your draft, then use the corrections policy to contact the editorial team.
Write a review or join the discussion