AI agents · OpenClaw · self-hosting · automation

Quick Answer

Best Open-Source Web Crawlers for LLMs and RAG 2026

Published:

The short answer

Start with Crawl4AI for RAG ingestion, switch to Scrapling for bot-protected sites, and use Crawlee or Scrapy when the crawl is large and long-running. What makes a crawler good for LLMs is not just fetching pages: it is rendering JavaScript, removing boilerplate and returning clean Markdown with links and metadata your chunker can use. Versions read from GitHub and PyPI on November 18, 2026:

CrawlerLicence · versionStarsLanguageLLM-ready outputAnti-bot / browsersBest for
Crawl4AIApache-2.0 · 0.9.5 (Oct 14, 2026)~87,500Python✅ Markdown, structured JSONPlaywright browsersMost RAG pipelines
FirecrawlAGPL-3.0 · firecrawl-py 4.57.0 (Nov 12, 2026)~203,000TypeScript (SDKs for Python, JS)✅ Markdown, JSONCloud has more than self-hostSame API locally and in the cloud
ScraplingBSD-3-Clause · 0.5.1 (Nov 4, 2026)~104,000Python✅ MarkdownStealth browsers, Cloudflare Turnstile handling, proxy rotationProtected sites, agents via MCP
CrawleeApache-2.0 · Python 1.11.2 (Nov 10) / JS 3.18.2~26,500 (JS) + ~9,700 (Py)Python, TypeScriptAdd your own extractionFingerprints, proxy and session poolsLarge, resumable crawls
ScrapyBSD-3-Clause · 2.20.0 (Nov 6, 2026)~65,500PythonAdd your own extractionMiddleware ecosystemHigh-volume static sites
SpiderMIT · v2.53.9 (Oct 30, 2026)~2,900RustMarkdown via featuresJS rendering only where neededMaximum speed

If you would rather pay per page than run browsers, see our comparison of web scraping and OCR APIs; for agents that need search results instead of crawls, see best web search APIs for AI agents.

1. Crawl4AI — best default for RAG

Crawl4AI was built for LLM ingestion: it renders pages with Playwright, strips boilerplate and returns Markdown or schema-driven JSON, so the output goes straight into a chunker. It runs as a Python library or a Docker server. The same crawler powers Crawl4AI Cloud, a hosted API with 1,000 free credits a month, so you can start hosted and move to self-hosting without rewriting extraction code. Recent commits landed November 17, 2026.

2. Firecrawl — best if you want a cloud fallback

Firecrawl is the most-starred project here and has a clean scrape/crawl/map/search API. The self-hosted server is AGPL-3.0; some cloud features are not in it. Firecrawl Cloud costs $0 for 1,000 credits a month, $19 for 5,000 (Hobby), $99 for 100,000 (Standard) and $399 for 500,000 (Growth), at one credit per page scraped. Pick it when you want the same interface for self-hosted bulk crawling and cloud bursts, and AGPL is acceptable.

3. Scrapling — best for difficult sites and agents

Scrapling combines fast HTTP fetching, stealth browsers, Cloudflare Turnstile and interstitial handling, adaptive element tracking that survives layout changes, concurrent spiders with pause/resume, and Markdown output. It also ships an MCP server and Agent Skill, so Claude Code or another agent can browse and extract through it directly.

4. Crawlee — best for large crawls

Crawlee, from Apify, provides request queues, persistent storage, session and proxy pools and human-like browser fingerprints in both Python and TypeScript. It is the most robust choice for crawls that run for hours and must resume after failures. It does not produce LLM-ready Markdown on its own; add an extraction step (for example readability plus a Markdown converter).

5. Scrapy — best for high-volume static sites

Scrapy remains the fastest Python option for sites that do not need JavaScript, with a mature middleware ecosystem for throttling, retries and pipelines. For JS-heavy pages, combine it with a browser plugin or use Crawl4AI instead.

6. Spider — best raw speed

Spider is a Rust crawling engine that streams pages as they arrive and renders JavaScript only on pages that need it. Use it when crawl throughput is the bottleneck; a hosted Spider Cloud handles proxies.

Crawling responsibly

Respect robots.txt and site terms, rate-limit per domain, identify your crawler, and keep the source URL and fetch date with every chunk so answers can cite it. After crawling, chunk along headings: see best chunking libraries for RAG.

Last verified: November 18, 2026. Versions and licences read from GitHub and PyPI; Firecrawl prices from firecrawl.dev/pricing.

Sources