Scrapling Review: Python Scraping With Spiders and MCP

TL;DR

Scrapling (D4Vinci/Scrapling) is a Python web scraping framework by Karim Shoair that puts four things in one library: fast HTTP requests that impersonate a browser’s TLS fingerprint, a stealth browser for JavaScript-heavy and protected pages, a Scrapy-like parser, and a concurrent spider framework with pause/resume. It also ships an MCP server so AI agents such as Claude or Cursor can scrape through it, and an Agent Skill that teaches coding agents its API.

Facts as of November 3, 2026:

• 93,687 GitHub stars and 9,838 forks. The repo was created on October 13, 2024, and it is on GitHub’s weekly Python trending list in the weeks after the v0.5 release.
• 4 open issues (issues only, not pull requests). That is unusually low for a project this size.
• License: BSD-3-Clause. Free, no hosted tier.
• Latest release: v0.5, published October 17, 2026, with many breaking changes and a dedicated migration guide.
• Requires Python 3.10+. pip install scrapling gives you only the parser; fetchers, spiders, the CLI and MCP need extras.

Our verdict: Scrapling is the most complete single-library answer to “I need to scrape this in Python” right now. One API covers a quick get, a rendered page, a 10,000-page crawl and an agent tool. Our container test passed all six steps in 44.6 seconds, including the README spider and a headless browser fetch. The cautions: v0.5 breaks old code (the plain Playwright backend is gone), the anti-bot features put the legal and ethical burden on you, and the README is half sponsor ads for proxy and CAPTCHA services.

Quick Reference

Field	Value
What it is	Python scraping framework: HTTP fetcher, stealth browser, parser, spiders, CLI, MCP server
Install	pip install "scrapling[fetchers]" then scrapling install for browsers; scrapling[all] for everything
Python	3.10 or newer
Browser backend	Patchright (a patched Playwright Chromium) since v0.5; Google Chrome optional
AI integration	MCP server (scrapling mcp, 16 tools per the docs), Agent Skill, page.markdown()
Docker	pyd4vinci/scrapling and ghcr.io/d4vinci/scrapling with browsers included
Latest release	v0.5 (October 17, 2026)
License	BSD-3-Clause
Price	Free

What Scrapling Actually Is

Most Python scraping stacks glue together requests, BeautifulSoup, Playwright and Scrapy. Scrapling’s pitch is one response type across all of them. A page from Fetcher.get() (plain HTTP), a page from StealthyFetcher.fetch() (a real browser) and a Response inside a spider callback all expose the same .css(), .xpath(), .find_all() and .markdown() methods. The parser uses Scrapy/Parsel’s pseudo-elements (::text, ::attr(href)).

The project has three layers:

1. Parsing. The Selector class works on any HTML string, with CSS, XPath, BeautifulSoup-style find_all, text and regex search, and DOM navigation (.parent, .next, .below_elements). Its distinctive feature is adaptive element tracking: you save an element’s fingerprint once, and when the site’s markup changes, Scrapling relocates it by similarity instead of failing on a dead selector.
2. Fetching. Fetcher and FetcherSession send HTTP requests through curl_cffi, impersonating a browser’s TLS fingerprint and headers, with HTTP/3 available. StealthyFetcher and StealthySession drive a Patchright Chromium with fingerprint spoofing, ad blocking (about 3,500 domains, per the README), optional DNS-over-HTTPS, remote browsers over CDP, and Cloudflare Turnstile handling when you pass solve_cloudflare=True.
3. Crawling. Spider gives you a Scrapy-style class with start_urls, async parse callbacks, concurrency limits, per-domain throttling, pause/resume through a crawldir, streaming, robots.txt support and built-in JSON, JSONL, CSV and XML export. Ready-made templates cover rule-based crawls, sitemaps, XML/CSV feeds, Shopify stores and whole-site Markdown corpora.

What Changed in v0.5

The October 17 release is the reason Scrapling is trending again, and it is a large one. The migration guide lists the breaking changes:

• The plain Playwright backend is gone. DynamicFetcher, DynamicSession, AsyncDynamicSession and PlayWrightFetcher are removed; every browser fetch now goes through the Patchright-based stealthy classes. If your callbacks import playwright.sync_api, change them to patchright.sync_api. Patchright evaluates JavaScript in an isolated context by default, so code that reads the page’s own JavaScript variables needs isolated_context=False.
• XHR capture is replaced by session network history. capture_xhr and response.captured_xhr are removed. You now set record_requests=True on a browser session and search session.network. The guide documents the limits: 1,000 requests retained by default, text bodies only, 1 MiB per response and 20 MiB in total.
• Login state you can save. save_state(path) and load_state(path) write cookies, local storage and IndexedDB to Playwright-compatible JSON, so a login survives between runs.
• Shadow DOM extraction with pierce_shadow=True, also exposed in the CLI.
• Crawl order control with crawl_strategy="bfs" or "dfs" on CrawlSpider and its subclasses.
• A rebuilt MCP server: form filling, clicks, keyboard input and dialogs in one ordered sequence, page snapshots with element roles and positions, element screenshots, custom JavaScript, network inspection and saved logins. Tool names changed, so agent configs that call the old names break.
• Spider behavior changes that can alter results: response.follow() no longer carries over POST bodies or query parameters, request fingerprints changed (old checkpoints and caches may not match), and a robots.txt fetch failure now blocks crawling for that domain when robots_txt_obey=True.

Getting Started

Install the fetchers, plus browsers if needed:

pip install "scrapling[fetchers]"
scrapling install     # downloads browsers and system dependencies

The smallest example from the README fetches a page over HTTP and parses it:

from scrapling.fetchers import Fetcher

page = Fetcher.get('https://quotes.toscrape.com/')
quotes = page.css('.quote .text::text').getall()
print(quotes)

For JavaScript-rendered pages, swap in the stealth browser. The parsing code does not change:

from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.fetch('https://quotes.toscrape.com/js/', wait_selector='.quote')
print(len(page.css('.quote')))

A spider that follows pagination, from the README:

from scrapling.spiders import Spider, Response

class QuotesSpider(Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]
    concurrent_requests = 10

    async def parse(self, response: Response):
        for quote in response.css('.quote'):
            yield {
                "text": quote.css('.text::text').get(),
                "author": quote.css('.author::text').get(),
            }
        next_page = response.css('.next a')
        if next_page:
            yield response.follow(next_page[0].attrib['href'])

result = QuotesSpider().start()
print(f"Scraped {len(result.items)} quotes")
result.items.to_json("quotes.json")

Pass crawldir="./crawl_data" to the constructor and Ctrl+C saves the queue; restarting with the same directory resumes. One spider can also route ordinary links through a fast FetcherSession and only protected pages through a lazily started AsyncStealthySession.

No code at all is needed for one-off extraction:

pip install "scrapling[shell]"
scrapling extract get 'https://quotes.toscrape.com/' quotes.md
scrapling extract fetch 'https://quotes.toscrape.com/js/' quotes.md --css-selector '.quote' --wait-selector '.quote'

For agents, install scrapling[ai] and register scrapling mcp in any MCP client. The docs describe one-shot tools (make_request, browser_fetch_once) and session tools (browser_open, browser_fetch, browser_screenshot and more).

Hands-on test (November 3, 2026)

We ran Scrapling in a fresh python:3.12-bookworm container on a GitHub-hosted runner (x86_64, 2 CPUs, 8 GB RAM, no GPU) against commit 95cf9019ab5a. We installed scrapling[all]==0.5, checked the version and CLI, ran the README’s HTTP example and pagination spider against quotes.toscrape.com (a public sandbox built for scraping practice), exported a page to Markdown with the CLI, and installed the browsers to fetch the site’s JavaScript-rendered variant. The verdict is passed: 6 of 6 steps, in 44.6 seconds in total.

Step	Result	Time
pip install "scrapling[all]==0.5"	Installed cleanly, including curl_cffi 0.17.0, browserforge and IPython	14.6 s
Version and scrapling --help	scrapling 0.5; CLI commands extract, install, mcp, shell	0.3 s
README Fetcher.get() example	Status 200, 10 quotes parsed, first quote printed	0.7 s
README spider with pagination	”Scraped 100 quotes” from 10 requests, all status 200, 0 failed, 0 blocked, 14.01 requests per second, 108,671 response bytes; quotes.json written (16,366 bytes)	1.2 s
scrapling extract get ... q.md	3,066-byte Markdown file with quotes, authors, tag links	0.5 s
scrapling install + StealthyFetcher.fetch() on /js/	Browsers and system dependencies installed; status 200, 10 quotes found after wait_selector	27.3 s

What worked: everything the README promises for the basic path, with no account, no API key and no manual browser setup. The spider’s end-of-run stats (requests per session, status codes, bytes per domain, blocked and robots-disallowed counts) are genuinely useful, and scrapling install handled Chromium and its Debian system packages on a stock image.

What surprised us: the HTTP fetcher’s log line shows referer: https://www.google.com/ on a plain Fetcher.get(). Scrapling adds browser-like headers, including a Google referer, by default. That helps against simple bot checks but is worth knowing if a site’s analytics or your own logs matter to you.

What this test does not tell you: quotes.toscrape.com has no bot protection, so we did not test Cloudflare solving, fingerprint quality or behavior against a real anti-bot vendor. We did not run the MCP server with an agent, try adaptive element tracking across a changed page, or measure memory on a long crawl. The parser benchmark numbers below are the project’s, not ours.

Performance Claims

The README publishes a parser benchmark (text extraction over 5,000 nested elements, averaged over 100+ runs, script in benchmarks.py). The project reports Scrapling at 1.99 ms, Parsel/Scrapy at 2.06 ms, raw lxml at 2.56 ms, PyQuery at 23.98 ms, Selectolax at 197.02 ms and BeautifulSoup with lxml at 1,562.1 ms. For similarity-based element finding it reports 2.3 ms against AutoScraper’s 12.58 ms.

Scrapling, Parsel and lxml are effectively tied because Scrapling parses with lxml. The gap to BeautifulSoup is real, but in most scrapers network time dominates parse time; our whole 10-page crawl took 1.2 seconds.

Things to Check Before You Adopt It

1. Upgrading from 0.4 is a migration, not a bump. Removed classes, renamed MCP tools, changed request fingerprints and a new robots.txt failure mode can all change your output. Pin scrapling==0.4.15 until you have worked through the migration guide.
2. Stealth is a legal question, not only a technical one. The README advertises bypassing “all types of Cloudflare’s Turnstile/Interstitial,” and the MCP docs say the server helps capable models get past CAPTCHAs. The project’s own disclaimer puts responsibility on you to follow terms of service, robots.txt and privacy law. Before pointing it at a site that blocks bots, decide whether you should be scraping it at all.
3. The MCP server gives an agent a real browser, with saved logins, form filling and custom JavaScript. Treat scraped pages as untrusted input and keep logged-in sessions away from agents unless you mean it.
4. The base install is parser-only. pip install scrapling and then from scrapling.fetchers import Fetcher raises ModuleNotFoundError. The README warns about this, but it catches people.
5. Sponsor-heavy documentation. Much of the README is ads for proxies and CAPTCHA-solving APIs. None are required.
6. One core maintainer. Karim Shoair drives the project. The low open-issue count suggests fast triage, but it is still a bus-factor question.

Who Should Use It

• Python developers who outgrow requests + BeautifulSoup and do not want to learn Scrapy’s project structure for a mid-sized job.
• Scrapy users who want a stealth browser or adaptive selectors. The scrapling_response decorator lets you use Scrapling’s parser inside existing Scrapy callbacks without a rewrite.
• Teams building RAG corpora. page.markdown() and the ContentSpider template turn a site into clean Markdown with no LLM in the loop.
• Agent builders who want a local, free scraping MCP server instead of a paid scraping API.

Skip it if you need a managed service with SLAs and someone else’s IP pool, or if your stack is TypeScript, where Crawlee and Playwright fit better.

Alternatives

Tool	Focus	Compared with Scrapling
Scrapy	Mature Python crawling framework	Bigger ecosystem and middleware; no built-in stealth browser; Scrapling can plug into it
Crawl4AI	LLM-oriented crawler with Markdown output	Similar RAG goal; Scrapling has the stronger general-purpose parser and spider API
CloakBrowser	Stealth Chromium build	A browser, not a framework; can serve as the browser Scrapling drives over CDP
browser-use	LLM agent that drives a browser	Model-driven navigation; Scrapling is deterministic code first, agent tools second
Firecrawl and other APIs	Hosted scrape-to-Markdown	No infrastructure to run, but paid per page; see our web scraping API comparison

Community Reactions

Scrapling has been announced on r/webscraping at major versions since its October 2024 launch. The v0.2.99 post in April 2025 drew 158 votes and 58 comments, and the February 2026 v0.4 announcement introduced the async spider framework that moved it into Scrapy’s territory. The README also links an independent review in The Web Scraping Club newsletter (November 2025). One telling detail: the README points users to paid sponsors for Akamai, DataDome, Kasada and Incapsula, a fair hint that Scrapling alone does not cover those.

FAQ

Is Scrapling free?

Yes. It is BSD-3-Clause licensed, with no paid tier. You pay only for any proxies or CAPTCHA services you choose to add.

Is Scrapling better than Scrapy?

It is different. Scrapy has the larger ecosystem and more production history. Scrapling adds a stealth browser, adaptive selectors and an MCP server with less boilerplate. Scrapy users can adopt its parser through the scrapling_response decorator without switching.

Does Scrapling bypass Cloudflare?

The project says StealthyFetcher with solve_cloudflare=True handles Cloudflare Turnstile and interstitial challenges, and v0.5 improved that solver. We did not test it. For Akamai, DataDome, Kasada and Incapsula, the README points to paid third-party services. Whether you should bypass a site’s bot protection is a terms-of-service and legal question you have to answer first.

How do I use Scrapling with Claude or Cursor?

Install pip install "scrapling[ai]", run scrapling install for the browsers, and add scrapling mcp as an MCP server in your client’s configuration. The agent then gets HTTP, browser, screenshot and session tools. There is also an Agent Skill that teaches coding agents the current API so the code they write matches v0.5.

What breaks when upgrading to Scrapling v0.5?

DynamicFetcher and the other plain-Playwright classes are removed (use StealthyFetcher/StealthySession), capture_xhr is replaced by record_requests=True and session.network, keep_cdata is gone, the CLI’s stealthy-fetch is now fetch, MCP tool names changed, and request fingerprints changed, so old crawl checkpoints may not resume cleanly.

Can Scrapling run in Docker?

Yes. The project publishes pyd4vinci/scrapling on Docker Hub and ghcr.io/d4vinci/scrapling with all extras and browsers. In our test, scrapling install set up browsers on a plain Debian Python image in 27.3 seconds, including the fetch that followed.

Bottom Line

Scrapling earns its stars. It replaces a four-library stack with one consistent API, the basic path works exactly as documented, and the spider framework is a credible, lighter alternative to Scrapy for most jobs. Version 0.5 is a serious upgrade, especially for agent use, but it is also a breaking one, so pin your version and read the migration guide. Use the stealth features deliberately: the code makes bypassing bot checks easy, and the responsibility for doing it stays with you.

Sources

• D4Vinci/Scrapling on GitHub: README, stars, license, issue counts, benchmarks
• Scrapling v0.5 release notes
• Migrating to v0.5 guide
• Scrapling MCP server documentation
• Scrapling documentation
• scrapling on PyPI
• Scrapling benchmarks.py
• r/webscraping: Scrapling v0.4 is here
• r/webscraping: Scrapling v0.2.99 website
• Trendshift: D4Vinci/Scrapling