Scraping Dashboard: Metrics, Errors, Speed in Real Time

Note: when your scraping processes millions of requests per day, you're blind without a dashboard. Errors accumulate, proxies go down, and you might not know for an hour. A scraping statistics dashboard solves this: see system health in real time, react to failures within minutes, optimize collection speed. We build such custom dashboards tailored to your data sources and infrastructure. With 5+ years in the market and 50+ successful projects, our experience guarantees stability. Get a project estimate in 1 day. Reach out for a consultation to discuss your task.

Key Metrics

Proxy Failure Solution and Critical Metrics

Picture this: a pool of 200 proxies, 15% of them periodically go down. Without monitoring, you lose up to 30% throughput. We embed the Proxy Health metric into your dashboard: when it drops below 95%, an alert is sent to Telegram. Analytics show that average recovery time drops from 40 to 8 minutes. Additionally, we configure automatic proxy rotation: when health falls below the threshold, the proxy is removed from the pool and its load redistributed to working addresses. This reduces lost requests by 20%. We also track response time and errors per proxy, allowing pinpoint replacement of problematic IPs. In a recent project for an e-commerce aggregator, this lowered the overall error rate from 12% to 3.5% within the first week. Contact our engineers to discuss your metrics.

From the set of metrics, pay special attention to Success Rate and Proxy Health. Success Rate indicates the proportion of successful responses—if it drops below 90%, the source is blocking requests or proxies are overloaded. Proxy Health tracks each proxy's operability: if health drops, the system automatically removes the proxy from rotation. Also important is Avg Response Time—a sharp increase often signals network issues or target server overload. We recommend configuring the dashboard to display all three metrics on the main screen.

Why TimescaleDB for Scraping Metrics?

For time-series data, specialized tools outperform standard PostgreSQL. TimescaleDB is a PostgreSQL extension with automatic time-based partitioning. Queries for weekly or monthly slices run fast even with 10 million rows. In one project, we stored 500 million records—aggregation took under 2 seconds. TimescaleDB also supports data compression, reducing storage size by 5–10 times without performance loss. Compared to plain PostgreSQL, TimescaleDB is 10x faster for time-series queries and saves 70% on storage costs.

# TimescaleDB (PostgreSQL extension) import psycopg2 CREATE TABLE scraper_metrics ( time TIMESTAMPTZ NOT NULL DEFAULT NOW(), scraper_id INTEGER, requests_ok INTEGER DEFAULT 0, requests_fail INTEGER DEFAULT 0, items_scraped INTEGER DEFAULT 0, avg_resp_ms FLOAT, proxy_used TEXT ); SELECT create_hypertable('scraper_metrics', 'time'); CREATE INDEX ON scraper_metrics (scraper_id, time DESC); 

API and Visualization

@app.get('/api/v1/stats/overview') async def get_overview(scraper_id: int, period: str = '24h'): interval = {'1h': '1 hour', '24h': '24 hours', '7d': '7 days'}[period] rows = await db.fetch(f''' SELECT time_bucket('5 minutes', time) AS bucket, SUM(requests_ok) AS ok, SUM(requests_fail) AS fail, SUM(items_scraped) AS items, AVG(avg_resp_ms) AS avg_ms FROM scraper_metrics WHERE scraper_id = $1 AND time > NOW() - INTERVAL '{interval}' GROUP BY bucket ORDER BY bucket ''', scraper_id) return { 'timeline': [dict(r) for r in rows], 'totals': { 'requests': sum(r['ok'] + r['fail'] for r in rows), 'success_rate': sum(r['ok'] for r in rows) / max(sum(r['ok'] + r['fail'] for r in rows), 1), 'items': sum(r['items'] for r in rows), } } 

import { LineChart, Line, XAxis, YAxis, Tooltip, ResponsiveContainer } from 'recharts'; function MetricsChart({ data }: { data: TimelinePoint[] }) { return ( <ResponsiveContainer width="100%" height={300}> <LineChart data={data}> <XAxis dataKey="bucket" tickFormatter={d => format(new Date(d), 'HH:mm')} /> <YAxis /> <Tooltip /> <Line dataKey="ok" stroke="#22c55e" name="Successful" strokeWidth={2} dot={false} /> <Line dataKey="fail" stroke="#ef4444" name="Failed" strokeWidth={2} dot={false} /> </LineChart> </ResponsiveContainer> ); } 

Alerts and Storage Comparison

How We Build Your Dashboard: Process

1. Analysis — We study your data sources and metric requirements. Identify critical KPIs. Estimate data volume and update frequency. Conduct interviews with your team to pinpoint bottlenecks.
2. Schema design — Choose tools (TimescaleDB, FastAPI, React). Design table structure and API endpoints. Plan for future scalability.
3. API implementation — Write backend for metric collection and aggregation. Handle up to 10,000 requests/second. Use async workers to minimize latency.
4. Frontend — Develop interface with charts and alerts. Use Recharts for interactive graphs. Add filters by time and scraper_id.
5. Testing — Load test up to 1000 requests/second. Fix bottlenecks. Perform UAT with your team.
6. Deployment — Deploy on your server or cloud. Monitor dashboard performance. Provide operational instructions.

What's Included and Timelines

• OpenAPI documentation
• Dashboard source code (backend and frontend)
• Deployment guide
• Team training (up to 2 hours)
• 3-month code warranty

Dashboard with TimescaleDB, REST API, and React visualization: 5–8 working days. We estimate your project in 1 day. By using our dashboard, clients reduce downtime costs by an average of $2,000 per month. Contact us for a consultation to get an architecture proposal for your load. Request a custom estimate in 1 day—send us your details.