Real-time WebSocket scraping for crypto exchanges and EVM chains

Imagine you trade on Binance using REST polling once per second. In that time, the price could have shifted 0.5%, and you missed an arbitrage opportunity. WebSocket subscriptions deliver events as they happen—latency drops from 500 ms to 10–50 ms. For price monitoring, order books, and on-chain events, the difference is critical.

Polling a REST API every N seconds is the wrong tool for event-driven tasks. With a 1-second poll interval, the average detection delay is 0.5 seconds. WebSocket subscriptions push events immediately; latency is only network-dependent (10–50 ms to the nearest exchange server). For price monitoring, order books, and on-chain events, the difference is fundamental. One of our clients cut latency from 800 ms to 30 ms by implementing WebSocket scraping for 50 pairs across 5 exchanges—saving up to 40% of lost profit.

Why WebSocket scraping beats REST polling for real-time data?

WebSocket cuts latency by 10x compared to REST polling (0.5 s → 50 ms)—saving up to 40% of lost profit. For crypto trading and DeFi bots, this difference is critical.

How to set up WebSocket connections to exchanges?

Each exchange has its own subscription protocol. Patterns are similar but details vary.

Binance: stream names via symbol@streamType

import asyncio import json import websockets async def binance_stream(symbols: list[str]): streams = '/'.join([f"{s.lower()}@trade" for s in symbols]) url = f"wss://stream.binance.com:9443/stream?streams={streams}" async with websockets.connect(url, ping_interval=20, ping_timeout=10) as ws: async for message in ws: data = json.loads(message) stream_data = data.get('data', data) yield { 'exchange': 'binance', 'symbol': stream_data['s'], 'price': float(stream_data['p']), 'amount': float(stream_data['q']), 'timestamp': stream_data['T'], 'is_buyer_maker': stream_data['m'], } 

Coinbase Advanced Trade: subscribe with channel and product_ids

subscribe_msg = { "type": "subscribe", "channel": "ticker", "product_ids": ["BTC-USD", "ETH-USD"], } 

Kraken

Uses subscription ID generation and has a specific response format with pairs in arrays. Details are in the official Kraken WebSocket API documentation.

Ethereum/EVM: WebSocket subscriptions via web3.py

On-chain events via WebSocket subscriptions to an Ethereum node (Alchemy, Infura, QuickNode, or your own node):

from web3 import AsyncWeb3, WebSocketProvider async def subscribe_to_transfers(token_address: str): w3 = AsyncWeb3(WebSocketProvider( "wss://eth-mainnet.g.alchemy.com/v2/YOUR_KEY" )) # ERC-20 Transfer event signature hash transfer_sig = w3.keccak(text="Transfer(address,address,uint256)").hex() subscription_id = await w3.eth.subscribe('logs', { 'address': token_address, 'topics': [transfer_sig] }) async for payload in w3.socket.process_subscriptions(): if payload['subscription'] == subscription_id: log = payload['result'] yield decode_transfer_log(log) 

Ethereum JSON-RPC WebSocket supports three subscription types: newHeads (new blocks), logs (contract events), and newPendingTransactions (mempool transactions). See the official Ethereum documentation for more details.

Why reconnect and staleness watchdog matter?

WebSocket connections can drop for various reasons: server timeout, network hiccups, exchange service restarts. A production system must recover automatically:

import asyncio import websockets from datetime import datetime class RobustWebSocketClient: def __init__(self, url: str, reconnect_delay: float = 1.0): self.url = url self.reconnect_delay = reconnect_delay self.max_reconnect_delay = 60.0 self.last_message_at = None self.stale_threshold = 30 # seconds without messages = staleness async def connect_with_retry(self, on_message, on_subscribe): delay = self.reconnect_delay while True: try: async with websockets.connect( self.url, ping_interval=20, ping_timeout=10, close_timeout=5, ) as ws: await on_subscribe(ws) delay = self.reconnect_delay # reset on success async for msg in ws: self.last_message_at = datetime.utcnow() await on_message(msg) except (websockets.ConnectionClosed, websockets.InvalidHandshake, OSError) as e: print(f"Connection error: {e}, reconnecting in {delay}s") await asyncio.sleep(delay) delay = min(delay * 2, self.max_reconnect_delay) async def staleness_watchdog(self): """Detect silent connection drop""" while True: await asyncio.sleep(10) if self.last_message_at: elapsed = (datetime.utcnow() - self.last_message_at).seconds if elapsed > self.stale_threshold: raise RuntimeError(f"Connection stale: {elapsed}s without data") 

Exponential backoff reconnection and a staleness watchdog are the minimum for industrial-grade scraping.

How to manage an order book via WebSocket?

Most exchanges send order book updates as incremental changes—only levels that changed. Maintaining a local order book:

from sortedcontainers import SortedDict class LocalOrderBook: def __init__(self): self.bids = SortedDict(lambda k: -k) # descending self.asks = SortedDict() # ascending self.last_update_id = 0 def apply_snapshot(self, snapshot: dict): self.bids.clear() self.asks.clear() for price, qty in snapshot['bids']: self.bids[float(price)] = float(qty) for price, qty in snapshot['asks']: self.asks[float(price)] = float(qty) self.last_update_id = snapshot['lastUpdateId'] def apply_update(self, update: dict): if update['u'] <= self.last_update_id: return # stale update, ignore for price, qty in update['b']: # bids p, q = float(price), float(qty) if q == 0: self.bids.pop(p, None) else: self.bids[p] = q for price, qty in update['a']: # asks p, q = float(price), float(qty) if q == 0: self.asks.pop(p, None) else: self.asks[p] = q self.last_update_id = update['u'] def best_bid(self) -> tuple[float, float]: k = next(iter(self.bids)) return k, self.bids[k] def best_ask(self) -> tuple[float, float]: k = next(iter(self.asks)) return k, self.asks[k] 

Important: on startup, get a snapshot via REST, then apply WebSocket updates starting with lastUpdateId > snapshotId. Updates before the snapshot are discarded; a gap in the U → u sequence requires a new snapshot.

Scaling: multiple pairs and exchanges

A single async event loop in Python handles 50–200 simultaneous WebSocket connections. For more, use multiple processes or a Go service (goroutines are significantly lighter than asyncio tasks).

Fanout results: processed messages are published to Redis Pub/Sub or Kafka for downstream consumers. The WebSocket handler should do minimal processing and publish quickly—heavy processing is done by a separate consumer.

Monitoring health

Metrics for each WebSocket connection: messages per second, reconnect count, last message timestamp, lag from exchange timestamp to processing timestamp. Use Grafana + Prometheus alerting on stale connections (no messages for active pair > 60 seconds).

What's included in WebSocket scraping setup

• Connection to exchanges / blockchain nodes via WebSocket (Binance, Coinbase, Kraken, Ethereum, Polygon, Solana, and others)
• Implementation of reconnect logic with exponential backoff and staleness watchdog
• Local order book aggregation with snapshot synchronization
• Publishing normalized data to Redis Pub/Sub or Kafka
• Monitoring and alerts (Grafana, Prometheus)
• Documentation of architecture and configuration
• Training your team on system operation

Our experience and guarantees

Over 5 years, we've completed 50+ real-time scraping projects for crypto exchanges, DeFi protocols, and NFT marketplaces. We guarantee stable operation, automatic recovery after failures, and 24/7 monitoring. We work with Ethereum, Binance, Polygon, Arbitrum, Solana, and other networks.

Setup of real-time scraping for 3–5 exchanges with monitoring of 20–50 pairs, reconnect logic, and publishing to Redis/Kafka takes 1–2 days. Contact us for a cost estimate. Order now to get a consultation.