The 9-Step Scrapy Loop Most Scripts Rebuild

Scrapy is a Python application framework for crawling sites and extracting structured data. Zyte maintains the project; 2.18.0 shipped 20 August 2026.

You write spiders and pipelines on top of an async Engine, not a one-file script. PyPI recorded about 4.5 million monthly downloads in early September 2026.

Say it SKRAY-peye (Wikipedia writes “SKRAY-peye”). The official FAQ compares pairing BeautifulSoup with Scrapy to pairing Jinja2 with Django: one parses, the other is the application. If you want the wider Python scraping stack, start from the Python web scraping hub, then come back here for the framework.

Key Takeaways

• Scrapy is a crawl framework: Engine, Scheduler, Downloader, Spider, Item Pipeline. BeautifulSoup is a parser you can still call inside a callback.
• New spiders use async def start(). Support for start_requests was removed in 2.16.
• Item pipelines clean, validate, dedupe, and persist. Feed exports already dump JSON, CSV, XML, and S3.
• Reach for Scrapy when the hard part is orchestration across many linked pages. A handful of pages once still belongs in requests plus BeautifulSoup.
• Scrapy does not execute JavaScript. scrapy-playwright is a download handler you opt into per request.
• In 2026, Firecrawl and Crawl4AI own one-shot LLM markdown. Scrapy still owns the crawl you run as a product.

What Is Scrapy?

Scrapy crawls websites and extracts structured data for mining, processing, and archival. You can also point it at APIs, or use it as a general-purpose crawler. The homepage tagline is “lean by design, extensible by nature.”

It is an application framework. You do not import scrapy and scrape a page in three lines. You scaffold a project, write a spider class, and let the Engine schedule requests.

You can still feed response.text into BeautifulSoup inside a callback when parsel gets awkward. That does not make the two tools peers.

Pablo Hoffman open-sourced Scrapy in 2008 after work at Mydeco (London) and Insophia (Montevideo). The GitHub repo dates to 22 February 2010; 1.0 landed in June 2015.

Zyte (then Scrapinghub) has been the steward since 2011. The license is BSD-3-Clause. Python 3.10 is the floor.

Live GitHub on 5 September 2026: 64,206 stars, 11,942 forks, 11,436 commits. Zyte counted 82 million cumulative downloads as of June 2025, and 575+ contributors.

Treat “the world’s most-used open source data extraction framework” as first-party marketing. It is a fair claim for production crawl frameworks. It is false if you read it as most-starred crawler in 2026 (Firecrawl sits at 176,877 stars).

This is tooling, not legal advice. Obey robots.txt and the site’s terms.

Why Scrapy Still Matters in 2026

The 2026 release train is not a fossil. 2.14.1 shipped 12 January. 2.15.0 (9 April) added an experimental reactor-less / httpx path.

2.16.0 (19 May) added Python 3.14 and Twisted 26.4+. 2.17.0 (7 July) added HTTP/2 and SOCKS on the httpx handler. 2.18.0 landed 20 August.

Search interest for the framework name is smaller than the new crawl APIs. That is not a eulogy. You still pick Scrapy when the crawl is a system you own.

Shane Evans, Zyte CEO and a Scrapy co-originator, put the 2025 stance this way: keep it a foundational framework, use Playwright for JavaScript, and let AI codegen sit on Scrapy rather than replace it. Mikhail Korobov (kmike), Zyte’s head of development, prefers extensibility over hard-coded features: retry and rate-limit control, an asyncio rewrite of the Twisted core, page objects (web-poet), spider templates.

On LinkedIn, Zyte still talks in spiders:

"Claude Code can write Python. But it doesn't know Scrapy patterns, page objects, or what makes a spider actually production-ready." (Zyte on LinkedIn, June 2026)

Generic Python from a model is not a production spider.

Install Scrapy

You need Python 3.10 or newer.

pip install scrapy

Confirm with scrapy version. Then:

scrapy startproject prices
cd prices

That scaffold is the first signal you left script-land: scrapy.cfg, settings.py, items.py, middlewares.py, pipelines.py, spiders/. Run spiders with scrapy crawl <name>.

The install page covers extras (Windows wheels, extra parsers).

How Scrapy Works: The Nine-Step Loop

The architecture page for 2.18.0 is a data-flow loop. Internals like download handlers are not a second course. Learn the nine steps, then write a spider.

• 1. The Engine gets initial Requests from the Spider.
• 2. The Engine schedules them and asks the Scheduler for the next Request.
• 3. The Scheduler returns the next Request.
• 4. The Engine sends it to the Downloader through Downloader Middlewares (process_request).
• 5. The Downloader fetches, and returns a Response through Downloader Middlewares (process_response).
• 6. The Engine sends the Response to the Spider through Spider Middleware (process_spider_input).
• 7. The Spider yields items and new Requests through Spider Middleware (process_spider_output).
• 8. The Engine sends items to Item Pipelines, and Requests back to the Scheduler.
• 9. Repeat until the Scheduler is empty.

That loop is why Scrapy is fast on multi-page crawls. Speed is overlapping requests, not faster parsing. Scrapy is written on Twisted.

It is non-blocking. Concurrent requests do not mean threads. A blocking call in a callback or pipeline (writing a huge spreadsheet with openpyxl) stalls the whole Engine.

Yield items and requests. Do not return a single item when you mean to keep crawling.

Engine, Scheduler, and Downloader

The Engine is the switchboard. The Scheduler is a priority queue. The Downloader speaks HTTP.

Downloader middleware is where proxies, retries, cookies, and robots.txt live. Spider middleware is depth, offsite filtering, referer. Extensions hook signals; they are not on the item data-flow path.

AutoThrottle and DOWNLOAD_DELAY plus concurrent-requests-per-domain are how you stay polite. Persistence is a setting, not a rewrite: scrapy crawl somespider -s JOBDIR=crawls/somespider-1 (clean shutdown only). See jobs.

Middleware Versus Your Spider

Spiders decide which requests to send and how to parse. Pipelines clean and persist.

Middleware rewrites requests and responses for every spider. Extensions react to signals.

Put the retry in middleware, not in parse. Put the PostgreSQL write in a pipeline, not in the spider. That split is what lets one project hold twenty spiders without becoming twenty scripts.

Browser rendering, monitoring, anti-ban, and page objects are extensions: scrapy-playwright, Spidermon, scrapy-zyte-api, scrapy-poet. The core stays lean.

Project Layout

scrapy startproject is the thesis in a tree.

prices/
  scrapy.cfg
  prices/
    __init__.py
    items.py
    middlewares.py
    pipelines.py
    settings.py
    spiders/

A throwaway script has none of this. You add it when the crawl has to survive a second site, a second developer, or a second month. Paweł Miech, a Scrapy contributor, warned at PyCon PL that the spider object also lets you build the opposite: a monster with navigation, yields, and field munging in one callback.

"If you're just going to use urllib and Beautiful Soup, well you need to probably handle redirects somehow. … You need to add some code to handle retrying because the request can fail. Duplicate filtering… caching… there's a really long list of things that urllib is not going to do for you. But at the same time, if your project is very simple and you don't need all those things, it should be fine." (Paweł Miech in "What's going on", PyCon PL, 20:35)

Redirects, retries, dupefilter, cache, robots, concurrency limits: the framework ships them so default Scrapy is not a denial-of-service client. A 40-line script does not need that list.

Scrapy Spiders: Requests, Callbacks, and start()

A spider is a class that defines how a site (or group of sites) is scraped: which requests to send, and how to parse responses into items and more requests.

The crawl loop inside the spider is smaller than the Engine loop. You iterate start() for initial requests (default: one Request per start_urls URL, callback parse). Scrapy downloads.

Your callback uses Selectors, yields items, and yields further Requests. Items go through pipelines and feed exports.

Required attribute: name. Keep it unique. Common practice is the domain without the TLD.

async def start()

Scrapy 2.13 introduced async def start() -> AsyncIterator and deprecated start_requests. Support for start_requests was removed in 2.16.

New spiders use async def start(). You can yield items from start() as well as requests.

allowed_domains (OffsiteMiddleware) changed in 2.18.0: live edits during a crawl now take effect. start_urls and custom_settings stay ordinary attributes. Do not copy a 2019 gist that still shows def start_requests.

To run Scrapy from a script instead of scrapy crawl, you need a Twisted reactor, or (TWISTED_REACTOR_ENABLED=False) an asyncio event loop. AsyncCrawlerProcess starts it. Experimental reactor-less mode landed in 2.15.

A Small Spider You Can Run

Skip another quotes.toscrape.com tour. The 2.16 start method and response.follow are the parts 2019 gists get wrong.

import scrapy


class TitlesSpider(scrapy.Spider):
    name = "titles"
    allowed_domains = ["books.toscrape.com"]

    async def start(self):
        yield scrapy.Request(
            "https://books.toscrape.com/",
            callback=self.parse,
        )

    def parse(self, response):
        for card in response.css("article.product_pod"):
            yield {
                "title": card.css("h3 a::attr(title)").get(),
                "price": card.css(".price_color::text").get(),
            }
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

Save it under spiders/titles.py. Run:

scrapy crawl titles -o titles.jsonl

-o titles.jsonl is a feed export. You did not write a JSON pipeline. response.css is parsel, and response.follow is the queue you used to hand-roll.

scrapy shell against books.toscrape.com is how you test selectors before you bake them into the spider.

CrawlSpider and Rules

CrawlSpider is the usual class for “regular websites” that you walk via rules, Rule, and LinkExtractor. First matching rule wins.

Use it when the site is a tree of listing pages. Keep extraction in callbacks, not in the rule list.

John Watson Rooney’s teaching path is still the honest one: write the 114-line requests script first, then let the framework absorb the boring parts.

"We've written something like 115, 114 lines of code. We do things like we export to CSV, we clean some data, we loop through different pages, we pull product information out. But what if we could do most of this with a framework? Because you think this is a lot of code that we're going to be rewriting next time we want to actually scrape a different site, and that's where frameworks are really good." (John Watson Rooney in "Scrapy in 30 Minutes", 0:08)

The rewrite pain is real. The framework tax is real too. Pay it when you will rewrite that script for the next site, or run it tomorrow.

Item Pipelines: Clean, Validate, Store

After a spider yields an item, pipeline components run sequentially. Typical jobs: strip residual HTML, validate required fields, drop duplicates, write a database.

The contract: process_item must return the item or raise DropItem. Dropped items skip later components. Optional open_spider / close_spider.

Any of these may be async def. In 2.18.0, open_spider may raise CloseSpider.

from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem


class PricePipeline:
    def process_item(self, item):
        adapter = ItemAdapter(item)
        price = adapter.get("price")
        if not price:
            raise DropItem("missing price")
        adapter["price"] = price.replace("£", "").strip()
        return item

Return the item. Mutate-and-forget is how the next component receives None.

Feed Exports vs Pipelines

Docs are explicit: if you want every scraped item in a JSON file, use Feed exports. JsonWriterPipeline in the docs is a teaching example. It is not the production path.

Feed exports already know JSON, JSON Lines, CSV, XML, and S3. Pipelines exist for work a file dump cannot do: drop invalids, normalize fields, hash for dedupe, write Postgres.

Activation is a setting, not a decorator:

ITEM_PIPELINES = {
    "prices.pipelines.PricePipeline": 300,
}

Lower integer runs first. Customary range is 0-1000. Confirm in the crawl log: [scrapy.middleware] INFO: Enabled item pipelines:.

If that line is missing, the class is not in ITEM_PIPELINES, or custom_settings overrode it.

Two pipelines (clean, then store) is the usual shape. Keep the spider stupid. Keep the pipeline boring.

On r/scrapy, the recurring “I stored the whole crawl in memory” failure shows up when learners skip pipelines and jobs.

u/wRAR_ (February 2026) put it as architecture, not style. Keeping intermediate data in memory is a bad idea except on small learning projects.

For a million-URL crawl, use JOBDIR so the queue lives on disk. A script holds one response. Scrapy will happily hold the frontier in RAM until you tell it not to.

When Scrapy Beats a Script

“Scrapy vs BeautifulSoup” is the comparison with search volume. It is the wrong comparison. BeautifulSoup turns a string of HTML into a tree.

It does not fetch, retry, throttle, follow links, or persist. The genuine comparison is requests + BeautifulSoup (you assemble the crawl) against Scrapy (the framework supplies it).

A crawl has five stages: schedule, fetch, parse, follow links, store. requests owns fetch. BeautifulSoup owns parse.

You hand-write the other three. Scrapy owns all five.

Ask two questions: how many pages, and how often.

John Watson Rooney draws the same line without the table:

"What are your goals for this project? Are you just grabbing the data and running, or will you need to run this daily for the foreseeable future, or are you managing a spider and a network of data pipelines? If it's the latter you'll certainly see benefits from using Scrapy. But if you answer no to any of those questions then perhaps writing your own solution in plain Python is the better option." (John Watson Rooney in "Scrapy is THE best", 4:21)

The Migration Tell

You have rebuilt Scrapy when the BeautifulSoup script grows a URL queue, a seen set, a time.sleep, a retry counter, and a thread pool. You have the parts. You do not have the tests.

You can mix. BeautifulSoup(response.text, "lxml") inside a callback is fine for find_parent / text navigation that parsel expresses badly. Do that on the tricky page, not on every page.

The reverse also works: parsel ships separately, so Scrapy’s Selector runs inside a plain requests script.

On r/webscraping, the sequential consensus is stable: HTTP plus a parser first, Scrapy when the crawl is the system, a browser only for pages that actually need JS.

u/jagdish1o1 (February 2026) put the large-project side in one line: nothing beats Scrapy on a dedicated crawl of a static catalog.

u/kubrador (January 2026) on a 500k-SKU marketplace: Scrapy plus a Redis queue and about 20 concurrent requests can finish in a day if the site does not rate-limit you into unplugging the router. The success condition is politeness, not peak RPS.

Stay on a Script

Stay on requests + BeautifulSoup when:

• You need a handful of pages once.
• The parse is a step inside a larger app, not a crawl you will rerun.
• Learning startproject costs more than the job. One operator’s teaching curve (15 July 2026): BeautifulSoup in about 15 minutes to first data; Scrapy about half a day to a working spider.
• The site already exposes a stable JSON API.
• The page is a JavaScript shell. Neither Scrapy nor BeautifulSoup will see the data.

FlyByAPIs ran 1,050 pages on books.toscrape.com (Python 3.12, M2 MacBook Air, three-run average).

Sync requests + BeautifulSoup: 105 seconds (~10 pages/s, 18 LOC). Twenty threads: 14 seconds (~75 pages/s, 35 LOC). Scrapy at default 16 concurrent: 11 seconds (~95 pages/s, 28 LOC).

Peak RAM was 48 / 61 / 86 MB.

That is one sandbox run. The finding is concurrency, not “11 versus 14” as a law. Below roughly 50 pages, the absolute gap is a few seconds.

Pick the tool on maintainability. Memory goes the other way on million-URL jobs: the script holds one response; Scrapy needs JOBDIR.

For scheduling the crawl you already have, Python automation covers cron, workers, and pipelines around the spider. Scrapy is the spider, not the whole plant.

JavaScript Pages and scrapy-playwright

When the payload only exists after a browser runs, Scrapy core will not help. The official plugin is scrapy-playwright (1,444 stars, BSD-3, Python 3.10+, Scrapy 2.7+, Playwright 1.40+).

It is a download handler. Pages that need JS still travel the normal Scrapy workflow: scheduler, items, pipelines.

Per-request opt-in:

yield scrapy.Request(
    url,
    meta={"playwright": True},
    callback=self.parse,
)

Unmarked requests use the regular HTTP handler. Do not render every URL.

Activation: set DOWNLOAD_HANDLERS for https to ScrapyPlaywrightDownloadHandler, and TWISTED_REACTOR to AsyncioSelectorReactor (the default in new projects since Scrapy 2.7). Set the reactor in settings.py or custom_settings before Twisted initializes. Late is too late.

On r/webscraping, the hybrid split is already production folklore. u/study_english_br (September 2025) on marketplaces: Mercado Livre currently HTTP-only with Scrapy; Amazon prices arrive via JavaScript.

The 2026 production shape is boring: Scrapy for the HTTP majority, Playwright for the JS minority. Do not start hybrid on day one.

Fifty concurrent Playwright tabs consume gigabytes; the same concurrency in HTTP is megabytes. Strip images, CSS, and fonts before you render. Treat browser workers as disposable.

Playwright and Selenium are browser drivers. Scrapy is an HTTP crawler. For the driver-versus-driver question, see Playwright vs Selenium.

Zyte’s July 2026 note on PLAYWRIGHT_BROWSER_PROVIDER (Camoufox / Patchright) is the missing-middle warning: stock Playwright Chromium builds are testing browsers that basic anti-bot already recognizes. A stealth browser is not a finished anti-bot stack.

Do not migrate to Scrapy to “beat Cloudflare.” A POST that works in requests can still 403 inside Scrapy’s downloader with the same headers. That is fingerprint, not payload.

When Scrapy Is the Wrong Tool in 2026

A BeautifulSoup-only fight reads as 2019. The commercial gravity in 2026 is agent APIs that return markdown.

Firecrawl (YC S22) is the loudest substitute: one API call to markdown or JSON, AGPL-3.0, 176,877 stars, $14.5 million Series A in August 2025 (SiliconANGLE puts total funding at $16.2 million).

Founders: Caleb Peffer, Eric Ciarla, Nicolas Silberstein Camara. First-party traction claims (ARR, “developers”) are self-reported. Do not treat them as a census.

Crawl4AI is Unclecode’s OSS asyncio crawler. It is aimed at markdown-for-RAG rather than a long-running structured crawl.

Crawlee (Apify; Prague, 2015) is the library you will hit even as a Python reader: Crawlee JS 25,663 stars, Crawlee Python 9,490. Apify is the hosted Actor marketplace.

X and LinkedIn talk those APIs, not spiders. That does not make Scrapy obsolete. It means the “scrape a site” demand now splits: agents want markdown; data teams still want item pipelines.

Gianfranco Gugino’s line on LinkedIn is the cleanest category split: scrapers are something you build and maintain; a web layer is something you call. Firecrawl is a web layer. Scrapy is a scraper you maintain.

When Scrapy wins that split: same sites every day, volume past a few hundred thousand pages a month, and full request-pipeline control.

You keep data on your infra, with tight database pipelines and 18 years of Stack Overflow. When Firecrawl wins: heterogeneous sites, no dedicated data engineer, JS-by-default, a prototype with a two-week half-life.

Zyte Scrapy Cloud (or self-hosted Scrapyd) is the hosted path when you outgrow scrapy crawl on a laptop.

Twisted is still the engine. Zyte announced an asyncio rewrite in 2025. It is not a Twisted-free default yet.

Common Scrapy Mistakes to Avoid

Shipping start_requests on a 2.18 Spider

start_requests was removed in 2.16. New spiders use async def start(). Copy-paste from a 2019 post, or from a 2026 blog that still shows def start_requests, will fail in a way that looks like “Scrapy is broken.” Read the spiders page for the current signature.

Writing a JSON-File Pipeline

Feed exports already write JSON, JSON Lines, CSV, XML, and S3. A custom JsonWriterPipeline is a teaching example.

Production file output is -o out.jsonl (or a feed URI in settings). Use a pipeline to drop, normalize, dedupe, or write a database.

Forgetting ITEM_PIPELINES or return item

The class does nothing until it is in ITEM_PIPELINES. custom_settings on one spider can override the project and silently disable it.

Check the log for Enabled item pipelines. If you mutate the item and forget return item, the next component gets None.

Blocking the Engine

Twisted is not threads. A blocking write in parse or process_item pauses every in-flight request.

Push heavy work off the Engine (or make the pipeline async def and use a real async driver). yield requests. Do not return when you mean to keep crawling.

Playwright on Every URL, or Scrapy as Anti-Bot

meta={"playwright": True} is an opt-in. Render the JS minority. A full browser against a marketplace will collapse RAM long before Scrapy “fixes” anti-bot.

Scrapy does not execute JavaScript on its own, and it does not beat Cloudflare by existing. Check DevTools → Network → XHR/Fetch first. Many “Playwright” pages are a JSON endpoint with extra steps.

On LinkedIn, Zenrows keeps repeating a related production bug: challenge pages that return HTTP 200 get stored as successful scrapes. Fail loud when the expected payload is missing.