scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
TL;DR · 30-second scan
scrapy (Python) — Scrapy, a fast high-level web crawling & scraping framework for Python.
You're a Python team scraping at scale and want middleware/pipeline patterns out of the box.
The Python scraping veteran. Mature ecosystem, plugins for everything (caching, proxies, middlewares), and a years-honed pipeline architecture. Steeper learning curve than the modern alternatives but lower long-term cost when crawls grow beyond a quick weekend script.
You're a Python team scraping at scale and want middleware/pipeline patterns out of the box.
You're scraping JS-heavy SPAs — Scrapy needs Playwright integration which is awkward; Crawlee is cleaner.
Add this badge to your README to show your project is curated on StackPicks. Free, lightweight (180×28 SVG), and gives your visitors a one-click way to see honest take + alternatives.
[](https://stackpicks.dev/repo/scrapy-scrapy)
<a href="https://stackpicks.dev/repo/scrapy-scrapy"><img src="https://stackpicks.dev/api/badge/scrapy-scrapy" alt="Featured on StackPicks" width="180" height="28" /></a>
Are you the maintainer of scrapy/scrapy? Add the badge and we'll feature your project in the next weekly newsletter (~2,000 builders).