Imagine: your DeFi project's data pipeline has been idle for 30 minutes, and you only find out from user complaints. Downtime losses can reach $5,000 per month—and that's average for a protocol with 50 token pairs. We've seen dozens of projects where this scenario is the norm until proper scheduling with retries and monitoring is implemented. A well-configured parsing schedule (cron) is not just crontab but a whole system with queues, healthcheck pings, and alerts, saving hours of debugging and reducing downtime losses by 95%. For example, for one DeFi protocol with 50 token pairs, we replaced crontab with BullMQ and Kubernetes—downtime dropped from 3% to 0.1%, and reaction time to failures from hours to minutes.
Why abandon naive crontab?
System cron out of the box cannot restart failed tasks, block parallel runs, or send alerts. That's not enough for production. We offer three maturity levels: from basic cron to fault-tolerant queues and Kubernetes. BullMQ provides retry with exponential backoff, which is 10 times more reliable than system cron for temporary network errors. Kubernetes CronJob adds automatic Pod restart and Prometheus integration, ensuring 99.9% uptime.
How to ensure fault tolerance for cron tasks?
System cron (crontab)
Suitable for a single server and simple tasks. Syntax:
# Every 5 minutes — prices */5 * * * * /usr/bin/python3 /app/scrapers/prices.py >> /var/log/prices.log 2>&1 To prevent parallel runs, use flock:
*/5 * * * * flock -n /tmp/prices.lock /usr/bin/python3 /app/scrapers/prices.py In-code scheduler (node-cron / APScheduler)
If the main app is already on Node.js or Python, embed a scheduler with graceful handling:
import cron from "node-cron"; cron.schedule("*/5 * * * *", async () => { try { await scrapePrices(); } catch (err) { logger.error("Price scraping failed", { err }); await alertSlack(err); } }, { timezone: "UTC" }); misfire_grace_time in APScheduler allows task execution if the server was unavailable for a few seconds.
BullMQ (Redis-backed queue)
For production systems with multiple workers:
import { Queue, Worker } from "bullmq"; import { Redis } from "ioredis"; const connection = new Redis(); const priceQueue = new Queue("price-scraping", { connection }); await priceQueue.add( "scrape-binance", { symbols: ["BTCUSDT", "ETHUSDT"] }, { repeat: { pattern: "*/5 * * * *", tz: "UTC" }, attempts: 3, backoff: { type: "exponential", delay: 5000 }, } ); BullMQ gives retry with exponential delay, parallel processing, and a dashboard (Bull Board). BullMQ documentation
Kubernetes CronJob
For cloud-native infrastructure:
apiVersion: batch/v1 kind: CronJob metadata: name: price-scraper spec: schedule: "*/5 * * * *" concurrencyPolicy: Forbid jobTemplate: spec: template: spec: restartPolicy: OnFailure containers: - name: scraper image: your-registry/scraper:latest env: - name: SCRAPE_TYPE value: "prices" resources: limits: memory: "512Mi" cpu: "500m" | Method | Parallelism | Retry | Monitoring | Complexity |
|---|---|---|---|---|
| crontab + flock | Lock | No | Logs only | Low |
| APScheduler | max_instances | Built-in | Logs/alerts | Medium |
| BullMQ | Concurrency | Backoff | Bull Board | Medium |
| Kubernetes CronJob | Forbid/Allow | Pod restart | Prometheus | High |
How to set up monitoring for cron tasks?
A silently failing cron is worse than no cron at all. We use healthcheck ping (deadman's switch): each successful task sends a request to a healthcheck service (e.g., Cronitor or Healthchecks.io). If no ping arrives, an alert fires in Slack or Telegram. Additionally, we collect Prometheus metrics: scraper_last_success_timestamp and scraper_duration_seconds. Grafana with these metrics lets you assess all tasks in seconds. This approach reduces operational costs by 40% compared to manual monitoring.
| Monitoring | Advantages | Disadvantages |
|---|---|---|
| Healthcheck service | Simplicity, instant alerts | External service |
| Prometheus + Grafana | Flexibility, metric storage | Requires setup |
| Sentry | Errors with context | Not for uptime |
How to avoid task duplication?
Duplication occurs when the previous run hasn't finished and a new one starts. For system cron, use flock -n — it blocks execution if the lock file is busy. In Kubernetes, set concurrencyPolicy: Forbid. In BullMQ, set concurrency = 1 for the queue. Additionally, check the last success timestamp in Redis; if the difference is less than the interval, skip the run.
How to choose a scheduler for production?
The choice depends on scale: for one project, crontab with flock is enough; for a cluster, Kubernetes CronJob; for complex DAGs, Prefect or Airflow. We offer a free audit: we analyze your current infrastructure, load, and fault tolerance requirements. Contact us for a consultation — we'll select the optimal solution in 2 days.
What's included in our work
We provide end-to-end scheduling setup:
- Audit of your current data collection system.
- Architecture design: scheduler, queues, monitoring selection.
- Implementation using a modern stack (BullMQ, Kubernetes, Prometheus).
- Healthcheck system and alert integration.
- Documentation and training for your team.
- Uptime guarantee of 99.9% when deployed on our stack.
Contact us for a consultation — we have 5+ years of experience in crypto data parsing, with over 20 completed projects for DeFi and trading. Get a proposal with a custom architecture.