Scraping via Scrapy (Python)

Implementation of Scraping via Scrapy (Python)

Scrapy is an industrial web scraping framework for Python. Not just a library, but a full architecture: built-in request queue, middleware system, pipeline for data processing, built-in robots.txt support, user-agent auto-rotation, response caching. If you need to collect data from hundreds of pages in parallel—Scrapy is the right choice.

Scrapy Architecture

Spider (traversal logic)
    ↓
Scrapy Engine
    ↓
Scheduler (URL queue)
    ↓
Downloader (HTTP requests)
    ↓ (via Downloader Middlewares)
Response → Spider
    ↓
Items → Item Pipeline
    ↓
Storage (DB, CSV, JSON, S3)

Each component is replaceable: can plug in custom queue (Redis via scrapy-redis), custom downloader (Playwright via scrapy-playwright), or custom pipeline.

Basic Spider

import scrapy

class CatalogSpider(scrapy.Spider):
    name = 'catalog'
    start_urls = ['https://example.com/catalog?page=1']

    def parse(self, response):
        for item in response.css('.product-card'):
            yield {
                'title': item.css('.title::text').get('').strip(),
                'price': item.css('.price::attr(data-value)').get(),
                'url': response.urljoin(item.css('a::attr(href)').get()),
            }

        next_page = response.css('a.next-page::attr(href)').get()
        if next_page:
            yield response.follow(next_page, self.parse)

Scaling via scrapy-redis

For distributed collection on multiple servers:

# settings.py
SCHEDULER = 'scrapy_redis.scheduler.Scheduler'
DUPEFILTER_CLASS = 'scrapy_redis.dupefilter.RFPDupeFilter'
REDIS_URL = 'redis://redis:6379'
SCHEDULER_PERSIST = True  # queue doesn't clear on restart

With scrapy-redis multiple workers read from shared Redis queue—horizontal scaling without code changes.

Middleware for Protection Bypass

class RotateUserAgentMiddleware:
    agents = [
        'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...',
        'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ...',
    ]

    def process_request(self, request, spider):
        request.headers['User-Agent'] = random.choice(self.agents)

Additionally: scrapy-rotating-proxies for automatic proxy rotation with status tracking per address.

Pipeline for PostgreSQL

class PostgreSQLPipeline:
    def open_spider(self, spider):
        self.conn = psycopg2.connect(DATABASE_URL)
        self.cur = self.conn.cursor()

    def process_item(self, item, spider):
        self.cur.execute(
            'INSERT INTO products (title, price, url) VALUES (%s, %s, %s) '
            'ON CONFLICT (url) DO UPDATE SET price = EXCLUDED.price',
            (item['title'], item['price'], item['url'])
        )
        self.conn.commit()
        return item

ON CONFLICT DO UPDATE solves deduplication at DB level without extra checks in code.

Monitoring and Statistics

Scrapy logs detailed statistics of each run: request count, items processed, errors, average response time. Via scrapy-prometheus these metrics export to Prometheus and visualize in Grafana.

Timeline

Spider for one site with pipeline and basic middleware: 3–5 days. Distributed system with Redis, monitoring, multiple source adapters: 10–15 days.