Crawlee
Crawlee fits JavaScript teams building crawlers that need request queues, stored output and browser automation.
Community support
Upvotes come from SearchEngineAnswer members. GitHub stars and reader reviews are counted separately.
Claim this tool if you maintain it.
- Best for
- Crawling
- Setup required
- Node.js; optional browser engine
- Costs to check
- Local compute or optional paid Apify hosting
- License in catalog
- Apache-2.0
GitHub repository
- GitHub stars
- 26,394
- Forks
- 1,747
- Last push
- 2026-11-09
- Primary language
- TypeScript
GitHub snapshot from . Stars count GitHub bookmarks. Reader ratings appear separately below.
How to install Crawlee
-
In a Node.js project, install Crawlee and Playwright if you plan to use the browser crawler.
npm install crawlee playwright - Follow the browser installation instructions for your environment. Start with a handler that stores the URL and the field you want to extract.
First tutorial for Crawlee
- Compare extraction from a static page with extraction from a page where JavaScript inserts the target field.
- Log which crawler mode produced each result. Verify that retries do not add duplicate rows.
Where Crawlee fits
Build a JavaScript or TypeScript crawler that needs to handle both raw HTML and browser-rendered pages.
- When to choose another tool
- Use a simpler HTML crawler for fields present in the initial response. Use Playwright directly when you only need a handful of scripted browser checks.
- First evaluation
- Choose one static page and one page with a field added after rendering. Check which crawler mode can extract each field, then log the URL, extraction result and any retry.
- How to interpret the result
- Budget for browser overhead. Check that retries do not create duplicate records or turn a missing field into a silent success.
Reviews and discussion
Reader reviews
Be the first to share your experience with this tool.
Discussion
Ask a question or share a setup tip.
Flag an outdated detail
Include the field that needs updating and a source link. Copy your draft, then use the corrections policy to contact the editorial team.
Write a review or join the discussion