← Tool directory
TOOL PROFILE

Spatie Crawler

Build a bounded PHP crawl for internal-link inventories or custom SEO checks.

Open source

Community support

Upvotes come from SearchEngineAnswer members. GitHub stars and reader reviews are counted separately.

Claim this tool if you maintain it.

Best for
Crawling
Setup required
A PHP project with Composer and a PHP version that meets the installed package's constraints.
Costs to check
Budget for engineering and crawl resources. Browser rendering has separate Chrome and Puppeteer requirements.
License in catalog
MIT

GitHub repository

GitHub stars
2,828
Forks
368
Last push
2026-11-05
Primary language
PHP

GitHub snapshot from . Stars count GitHub bookmarks. Reader ratings appear separately below.

How to install Spatie Crawler

  1. Check the repository's composer.json for the required PHP version and install the package with composer require spatie/crawler.
  2. Load the Composer autoloader and create a small script using the documented Spatie Crawler class.

First tutorial for Spatie Crawler

  1. Use Crawler::create with fake responses for a homepage and its linked About page. Call foundUrls to collect the discovered addresses.
  2. Check that the About URL appears. Before trying a real site, use a shallow depth and limit the crawl to internal URLs.

Official installation and usage documentation ↗

Where Spatie Crawler fits

Use this when you need a crawl integrated with PHP code and want to control how discovered URLs are handled.

When to choose another tool
Choose a standalone desktop crawler if your team needs an interactive report without maintaining PHP code.
First evaluation
Use the documented fake response map with a homepage that links to a second known internal URL.
How to interpret the result
A passing fixture confirms your link discovery logic. Real sites still require checks for crawl limits and JavaScript content.

Reviews and discussion

Reader reviews

Be the first to share your experience with this tool.

Discussion

Ask a question or share a setup tip.

Write a review or join the discussion

Reviews should describe your own experience. All submissions are moderated.

Flag an outdated detail

Include the field that needs updating and a source link. Copy your draft, then use the corrections policy to contact the editorial team.