← Tool directory
TOOL PROFILE

Crawler Commons

Use Java components for robots.txt and sitemap processing in a crawl pipeline.

Open source

Community support

Upvotes come from SearchEngineAnswer members. GitHub stars and reader reviews are counted separately.

Claim this tool if you maintain it.

Best for
Crawling
Setup required
A Java project and Maven for the upstream dependency setup.
Costs to check
Budget for integration work and JVM runtime. Local parser tests require no hosted account.
License in catalog
Apache-2.0

GitHub repository

GitHub stars
260
Forks
93
Last push
2026-10-29
Primary language
Java

GitHub snapshot from . Stars count GitHub bookmarks. Reader ratings appear separately below.

How to install Crawler Commons

  1. Add the documented com.github.crawler-commons crawler-commons dependency to the Maven project.
  2. Resolve the Maven dependencies and prepare a local sitemap fixture with two known URL entries.

First tutorial for Crawler Commons

  1. Use the package's sitemap parser to read the fixture and extract the two locations.
  2. Compare the parsed locations with your expected list and repeat with one entry removed.

Official installation and usage documentation ↗

Where Crawler Commons fits

Use this when a Java pipeline needs dedicated parser components and explicit test fixtures.

When to choose another tool
Use a complete crawler when you need a ready-made crawl report instead of parser building blocks.
First evaluation
Parse a small sitemap with two known URLs and compare the entries against an expected list.
How to interpret the result
Successful parsing establishes what the file lists. It does not prove that a URL is reachable or indexed.

Reviews and discussion

Reader reviews

Be the first to share your experience with this tool.

Discussion

Ask a question or share a setup tip.

Write a review or join the discussion

Reviews should describe your own experience. All submissions are moderated.

Flag an outdated detail

Include the field that needs updating and a source link. Copy your draft, then use the corrections policy to contact the editorial team.