Crawler Commons
Use Java components for robots.txt and sitemap processing in a crawl pipeline.
Community support
Upvotes come from SearchEngineAnswer members. GitHub stars and reader reviews are counted separately.
Claim this tool if you maintain it.
- Best for
- Crawling
- Setup required
- A Java project and Maven for the upstream dependency setup.
- Costs to check
- Budget for integration work and JVM runtime. Local parser tests require no hosted account.
- License in catalog
- Apache-2.0
GitHub repository
- GitHub stars
- 260
- Forks
- 93
- Last push
- 2026-10-29
- Primary language
- Java
GitHub snapshot from . Stars count GitHub bookmarks. Reader ratings appear separately below.
How to install Crawler Commons
- Add the documented com.github.crawler-commons crawler-commons dependency to the Maven project.
- Resolve the Maven dependencies and prepare a local sitemap fixture with two known URL entries.
First tutorial for Crawler Commons
- Use the package's sitemap parser to read the fixture and extract the two locations.
- Compare the parsed locations with your expected list and repeat with one entry removed.
Where Crawler Commons fits
Use this when a Java pipeline needs dedicated parser components and explicit test fixtures.
- When to choose another tool
- Use a complete crawler when you need a ready-made crawl report instead of parser building blocks.
- First evaluation
- Parse a small sitemap with two known URLs and compare the entries against an expected list.
- How to interpret the result
- Successful parsing establishes what the file lists. It does not prove that a URL is reachable or indexed.
Reviews and discussion
Reader reviews
Be the first to share your experience with this tool.
Discussion
Ask a question or share a setup tip.
Flag an outdated detail
Include the field that needs updating and a source link. Copy your draft, then use the corrections policy to contact the editorial team.
Write a review or join the discussion