THE DIRECTORY

SEO tool directory

Compare crawlers, audit tools, and AI search integrations by task and access.

22 tools

advertools

This is a strong starting point for SEO analysts who already work in Python.

CrawlingOpen source

Scrapy

Pick Scrapy when the audit needs custom extraction rules, crawl queues and a maintained scraping framework rather than a finished SEO report.

CrawlingOpen source

Crawlee

Crawlee fits JavaScript teams building crawlers that need request queues, stored output and browser automation.

CrawlingOpen source

LibreCrawl

Crawl a site from a desktop workflow and export technical findings for review.

CrawlingOpen source

Trafilatura

Extract main article text from HTML for content inventories and template quality checks.

CrawlingOpen source

uro

Reduce repetitive URL variants when building a small sample for a crawl or audit.

CrawlingOpen source

Spatie Crawler

Build a bounded PHP crawl for internal-link inventories or custom SEO checks.

CrawlingOpen source

Puppeteer

Automate browser checks of rendered metadata and links on a defined set of pages.

CrawlingOpen source

Spatie Robots Txt

Check crawl rules for specified paths and user agents from PHP.

CrawlingOpen source

Colly

Build a Go crawler that visits selected URLs and collects page or link evidence.

CrawlingOpen source

Crawler Commons

Use Java components for robots.txt and sitemap processing in a crawl pipeline.

CrawlingOpen source

Crawl4AI

Extract browser-rendered page content for repeatable inspection in Python.

CrawlingOpen source

Courlan

Filter and normalize candidate URLs before a crawl or content extraction run.

CrawlingOpen source

Sitemapper

Read sitemap XML and collect listed URLs from Node.js.

CrawlingOpen source

CrawlObserver

Run technical site crawls and inspect metadata, links, and indexability in a local interface.

CrawlingOpen source

Protego

Test how a robots.txt file applies to a specified user agent and URL.

CrawlingOpen source

gocrawl

Run bounded technical SEO crawls and export findings as JSON, CSV, or HTML.

CrawlingOpen source

Browsertrix Crawler

Capture browser-rendered pages in a replayable archive before a website migration.

CrawlingOpen source

goquery

Extract titles and canonical links from saved HTML with CSS selectors in a Go audit script.

CrawlingOpen source

ReplayWeb.page

Replay a saved site capture before a migration and inspect what the archive actually preserved.

CrawlingOpen source

Compare tools