Industrial Web Scraping System Development
A client came to us with a task: daily collect prices and stock from 20 competitor sites. A week after launching the first script using requests, all IPs were blocked by Cloudflare protection. A month later the markup changed — data collection stopped entirely. That's a typical failure without an industrial approach. We build systems that don't break: priority scheduler, residential proxy rotation, anti-bot bypass, normalization, and monitoring. Our team has delivered 15+ projects for e-commerce, aggregators, and research tasks. Budget savings on data collection typically reach up to 60%, with system payback in 2–3 months. Typical project costs range from $3,000 for a simple parser to $20,000 for a full multi-source system. For instance, one client saved $5,000 monthly on data acquisition costs. If you need a reliable data collection system, contact us for a consultation.
How to Bypass Cloudflare Bot Management?
The toughest case is Cloudflare with Bot Fight Mode. Solution: Playwright with a real browser fingerprint, bypass via puppeteer-extra-plugin-stealth, mouse movement simulation through CDP. For extra resilience — IP rotation through residential proxies from BrightData or Oxylabs.
| Protection | Bypass Method |
|---|---|
| Rate limiting | Adaptive delays, IP distribution |
| CAPTCHA (reCAPTCHA v2/v3) | 2captcha/Anti-Captcha API or custom model training |
| Cloudflare Bot Management | Playwright with real fingerprint, cyclic TLS fingerprint rotation |
| JavaScript challenges | Headless browser with full JS execution |
| Honeypot links | Filter invisible elements before crawling |
| IP reputation blocks | Residential proxy (BrightData, Oxylabs, Smartproxy) |
Why Does the Parser Break in a Month?
Websites change structure every 3–4 weeks. Without a monitoring system, you only find out about the break when data stops updating. We embed automatic checks: DOM schema comparison with baseline, successful extraction rate, fixture tests. A typical stable operation rate is 95%+ successful parses.
Architecture of a Scalable Scraping System
Scheduler and Task Queue
Celery with Redis or RabbitMQ is a proven choice. Each URL is a task with priority, retry policy, and TTL. Scrapy-cluster or a custom orchestrator coordinates workers. We use Python 3.12, Celery 5.3, Redis 7.
Page Fetcher
Two modes:
- Static —
httpxwith async, connection pooling, keep-alive - JavaScript rendering — Playwright 1.40 (preferred) or Puppeteer, headless Chromium with profile management
Identity Rotation
Proxy pool (residential or datacenter), User-Agent rotation from real fingerprint datasets, random delays with normal distribution, cookie session management.
Data Extraction
CSS selectors or XPath for stable structures. For complex logic — parsel (wrapper over lxml). If the structure is unstable — LLM extraction via OpenAI or local Ollama with few-shot prompts.
Storage and Normalization
Raw HTML in S3/MinIO for reprocessing. Extracted data — PostgreSQL 16 or ClickHouse (for analytics on billions of records). Deduplication by URL hash + content hash.
Comparison: Celery vs Argo Workflows
| Criterion | Celery | Argo Workflows |
|---|---|---|
| Setup complexity | Low (Python stack) | Medium (Kubernetes) |
| Scaling | 100+ workers | 1000+ workers |
| Failure response time | Minutes | Seconds |
| Community | Large | Growing |
For most projects, Celery is 3 times better than Argo for implementation speed and 5 times better in cost-effectiveness, while still handling over 100 concurrent workers.
Architecture for High-Load Scraping
Architecture diagram (click to expand)
[Scheduler] -> [Redis Queue] -> [Fetcher Workers x N] | [Parser Workers x M] | [Raw Store S3] + [DB Writer] | [Monitor / Dashboard] Fetcher and Parser are separate workers. Fetcher is I/O bound (100+ async tasks per process), Parser is CPU bound (1 process per core).
How to Set Up a Parser in 5 Steps
- Source analysis: study target site structure, determine data volume and update frequency.
- Stack selection: decide if JS rendering is needed, what proxy type, which database.
- Extraction implementation: write selectors or XPath for each field, test on fixtures.
- Monitoring setup: add alerts for drop in successful parse rate and DOM changes.
- Deploy and trial: deploy on server, run test period of 2–3 days.
Legal and Ethical Aspects
Before launch: check robots.txt, analyze site ToS, assess load. For public data this is usually acceptable. For restricted sections — permission required. We always advise on legal risks.
Process and Timelines
What's Included in the Work
- System architecture for your sources
- Code implementation with documentation
- Monitoring and alert setup
- Your team training
- One month post-launch support
- Stable operation guarantee (per SLA)
Implementation Timelines
| Stage | Timeline |
|---|---|
| Basic parser for one site | 3–5 days |
| Queue + proxy rotation + retry | 5–7 days |
| JS rendering + anti-bot bypass | 7–14 days |
| Monitoring, normalization, storage | 5–10 days |
| Full system for 10+ sources | 4–8 weeks |
Use Cases and Maintenance
Competitor monitoring: prices, assortment, availability — hourly collection with history. Up to 60% time savings for your analysts.
Ad aggregation: OLX, Avito-like platforms — tens of thousands of records daily, deduplication, geocoding.
Research tasks: dataset collection for ML, sentiment monitoring, SEO position analysis. Content projects: news syndication, job aggregation, open-source catalogs.
System Maintenance
A well-designed system is not a one-time development but an infrastructure with a lifecycle. Allocate 20% of development time per year for support. We guarantee rapid response to failures.
We'll assess your project in 1 day — write to us. Order a turnkey scraping system development and get an architecture consultation.