Scraping Failure Alerts: Email and Telegram Notifications

Scraping Failure Alerts: Why and When?

Parser went down at night — by morning data is stale and no one knows why. We've seen this in 80% of projects where monitoring was limited to logs. An alert system solves it: the right person gets a notification about a parsing failure the moment it occurs — via Telegram or email, with enough context for diagnosis. Without such notifications, an engineer spends hours hunting for the cause while client data remains outdated.

Automatic alerts aren't a luxury; they're a necessity for any scraping pipeline. Lack of monitoring leads to data loss and reputational risk. For example, a change in the target site's structure can go unnoticed for days until an empty result piles up. Our turnkey implementation — from design to deployment — lets you quickly add alerts to any parser on any stack, saving up to 10 hours of troubleshooting per month.

Which Events Require an Alert?

Not every error is a failure. A single timeout is normal — the worker will retry. The alert system triggers on:

• Task exhausted all retries (moved to DLQ or failed finally)
• Worker crashed (process crash, OOM)
• Error rate exceeded threshold in the last 15 minutes (e.g. >20%)
• Scraping a site didn't finish within expected time (watchdog timeout)
• Page structure changed — parser returns empty data

Which Notification Channel: Telegram or Email?

We usually recommend Telegram for P1-level alerts (site down) and email for less urgent events. In our experience, a hybrid scheme cuts engineer response time by 3–4x.

Why Telegram Is Best for Critical Failures?

Telegram messages arrive 10–30 times faster than email via SMTP and aren't subject to spam filters. In our projects, the time from failure to alert receipt via Telegram never exceeds 2 seconds. For tasks where every second of downtime costs money, Telegram is the only choice. According to the Telegram Bot API documentation, messages are delivered almost instantly.

Telegram: Bot Setup and Alert Sending

Example code for sending notification via Telegram Bot API:

import httpx import textwrap async def send_telegram_alert(bot_token: str, chat_id: str, event: dict): text = textwrap.dedent(f""" 🔴 <b>Parsing Failure</b> <b>Site:</b> {event['site_name']} <b>URL:</b> <code>{event['url']}</code> <b>Error:</b> {event['error_type']} <b>Message:</b> <code>{event['error_message'][:300]}</code> <b>Attempts:</b> {event['attempts']} <b>Time:</b> {event['timestamp']} """).strip() async with httpx.AsyncClient() as client: await client.post( f"https://api.telegram.org/bot{bot_token}/sendMessage", json={"chat_id": chat_id, "text": text, "parse_mode": "HTML"}, timeout=10, ) 

Email: SMTP and SendGrid Setup

For email, you can use SMTP (smtplib with TLS) or SendGrid for better deliverability. Example with SendGrid:

from sendgrid import SendGridAPIClient from sendgrid.helpers.mail import Mail def send_email_alert(to_email: str, event: dict): message = Mail( from_email='[email protected]', to_emails=to_email, subject=f"[Scraping] Failure: {event['site_name']}", html_content=render_alert_template(event), ) sg = SendGridAPIClient(api_key=SENDGRID_API_KEY) sg.send(message) 

How Deduplication Prevents Alert Spam?

Without deduplication, a mass failure (proxy provider down) would trigger 500 emails per minute. The solution is grouping by key with a cooldown. One alert per error type per 30 minutes is a reasonable balance between informativeness and noise. In our practice, this reduces notifications by 95% while preserving critical information.

def should_send_alert(site_id: int, error_type: str, cooldown_minutes: int = 30) -> bool: key = f"alert_sent:{site_id}:{error_type}" if redis.exists(key): return False redis.setex(key, cooldown_minutes * 60, "1") return True 

Example configuration with Redis:

import redis import os r = redis.Redis.from_url(os.environ["REDIS_URL"]) COOLDOWN = 30 # minutes def should_send_alert(site_id, error_type): key = f"alert_sent:{site_id}:{error_type}" if r.exists(key): return False r.setex(key, COOLDOWN * 60, "1") return True 

How to Set Up Telegram Notifications in 15 Minutes?

1. Create a bot via @BotFather and get the token.
2. Determine chat_id (use @userinfobot).
3. Integrate the send_telegram_alert function into your parser.
4. Trigger it on failure events.
5. Test sending.

Implementation Process: From Analysis to Deployment

• Design alert scheme (channels, thresholds, cooldown).
• Develop notification code (Telegram bot / SendGrid / SMTP).
• Implement deduplication with Redis or in-memory.
• Integrate with your parser (webhook or API).
• Documentation and team training.
• Support for 2 weeks after delivery.

Timeline and Cost

Basic solution (Telegram + email with deduplication) — from 1 to 2 business days. If integration with existing monitoring or custom rules is needed — up to 5 business days. Cost is calculated individually based on complexity: contact us for a free project estimate.

Experience and Guarantees

We've built monitoring systems for projects with 10 million requests per day. Over 5 years of experience in scraping and 50+ successful implementations ensure that alerts won't miss a critical failure. Order an alert system implementation — and always stay informed about your scraping status.