Spatie Crawler
Build a bounded PHP crawl for internal-link inventories or custom SEO checks.
Community support
Upvotes come from SearchEngineAnswer members. GitHub stars and reader reviews are counted separately.
Claim this tool if you maintain it.
- Best for
- Crawling
- Setup required
- A PHP project with Composer and a PHP version that meets the installed package's constraints.
- Costs to check
- Budget for engineering and crawl resources. Browser rendering has separate Chrome and Puppeteer requirements.
- License in catalog
- MIT
GitHub repository
- GitHub stars
- 2,828
- Forks
- 368
- Last push
- 2026-11-05
- Primary language
- PHP
GitHub snapshot from . Stars count GitHub bookmarks. Reader ratings appear separately below.
How to install Spatie Crawler
- Check the repository's composer.json for the required PHP version and install the package with composer require spatie/crawler.
- Load the Composer autoloader and create a small script using the documented Spatie Crawler class.
First tutorial for Spatie Crawler
- Use Crawler::create with fake responses for a homepage and its linked About page. Call foundUrls to collect the discovered addresses.
- Check that the About URL appears. Before trying a real site, use a shallow depth and limit the crawl to internal URLs.
Where Spatie Crawler fits
Use this when you need a crawl integrated with PHP code and want to control how discovered URLs are handled.
- When to choose another tool
- Choose a standalone desktop crawler if your team needs an interactive report without maintaining PHP code.
- First evaluation
- Use the documented fake response map with a homepage that links to a second known internal URL.
- How to interpret the result
- A passing fixture confirms your link discovery logic. Real sites still require checks for crawl limits and JavaScript content.
Reviews and discussion
Reader reviews
Be the first to share your experience with this tool.
Discussion
Ask a question or share a setup tip.
Flag an outdated detail
Include the field that needs updating and a source link. Copy your draft, then use the corrections policy to contact the editorial team.
Write a review or join the discussion