Imagine you trade on Binance using REST polling once per second. In that time, the price could have shifted 0.5%, and you missed an arbitrage opportunity. WebSocket subscriptions deliver events as they happen—latency drops from 500 ms to 10–50 ms. For price monitoring, order books, and on-chain events, the difference is critical.
Polling a REST API every N seconds is the wrong tool for event-driven tasks. With a 1-second poll interval, the average detection delay is 0.5 seconds. WebSocket subscriptions push events immediately; latency is only network-dependent (10–50 ms to the nearest exchange server). For price monitoring, order books, and on-chain events, the difference is fundamental. One of our clients cut latency from 800 ms to 30 ms by implementing WebSocket scraping for 50 pairs across 5 exchanges—saving up to 40% of lost profit.
| Parameter | REST Polling | WebSocket |
|---|---|---|
| Event latency | 500 ms – 2 s | 10–50 ms |
| Server load | High (N requests/min) | Low (one connection) |
| Reaction to changes | Delayed, possible misses | Instant, all events in sequence |
| Implementation complexity | Low | Medium, requires reconnect logic |
Why WebSocket scraping beats REST polling for real-time data?
WebSocket cuts latency by 10x compared to REST polling (0.5 s → 50 ms)—saving up to 40% of lost profit. For crypto trading and DeFi bots, this difference is critical.
How to set up WebSocket connections to exchanges?
Each exchange has its own subscription protocol. Patterns are similar but details vary.
Binance: stream names via symbol@streamType
import asyncio
import json
import websockets
async def binance_stream(symbols: list[str]):
streams = '/'.join([f"{s.lower()}@trade" for s in symbols])
url = f"wss://stream.binance.com:9443/stream?streams={streams}"
async with websockets.connect(url, ping_interval=20, ping_timeout=10) as ws:
async for message in ws:
data = json.loads(message)
stream_data = data.get('data', data)
yield {
'exchange': 'binance',
'symbol': stream_data['s'],
'price': float(stream_data['p']),
'amount': float(stream_data['q']),
'timestamp': stream_data['T'],
'is_buyer_maker': stream_data['m'],
}
Coinbase Advanced Trade: subscribe with channel and product_ids
subscribe_msg = {
"type": "subscribe",
"channel": "ticker",
"product_ids": ["BTC-USD", "ETH-USD"],
}
Kraken
Uses subscription ID generation and has a specific response format with pairs in arrays. Details are in the official Kraken WebSocket API documentation.
Ethereum/EVM: WebSocket subscriptions via web3.py
On-chain events via WebSocket subscriptions to an Ethereum node (Alchemy, Infura, QuickNode, or your own node):
from web3 import AsyncWeb3, WebSocketProvider
async def subscribe_to_transfers(token_address: str):
w3 = AsyncWeb3(WebSocketProvider(
"wss://eth-mainnet.g.alchemy.com/v2/YOUR_KEY"
))
# ERC-20 Transfer event signature hash
transfer_sig = w3.keccak(text="Transfer(address,address,uint256)").hex()
subscription_id = await w3.eth.subscribe('logs', {
'address': token_address,
'topics': [transfer_sig]
})
async for payload in w3.socket.process_subscriptions():
if payload['subscription'] == subscription_id:
log = payload['result']
yield decode_transfer_log(log)
Ethereum JSON-RPC WebSocket supports three subscription types: newHeads (new blocks), logs (contract events), and newPendingTransactions (mempool transactions). See the official Ethereum documentation for more details.
Why reconnect and staleness watchdog matter?
WebSocket connections can drop for various reasons: server timeout, network hiccups, exchange service restarts. A production system must recover automatically:
import asyncio
import websockets
from datetime import datetime
class RobustWebSocketClient:
def __init__(self, url: str, reconnect_delay: float = 1.0):
self.url = url
self.reconnect_delay = reconnect_delay
self.max_reconnect_delay = 60.0
self.last_message_at = None
self.stale_threshold = 30 # seconds without messages = staleness
async def connect_with_retry(self, on_message, on_subscribe):
delay = self.reconnect_delay
while True:
try:
async with websockets.connect(
self.url,
ping_interval=20,
ping_timeout=10,
close_timeout=5,
) as ws:
await on_subscribe(ws)
delay = self.reconnect_delay # reset on success
async for msg in ws:
self.last_message_at = datetime.utcnow()
await on_message(msg)
except (websockets.ConnectionClosed,
websockets.InvalidHandshake,
OSError) as e:
print(f"Connection error: {e}, reconnecting in {delay}s")
await asyncio.sleep(delay)
delay = min(delay * 2, self.max_reconnect_delay)
async def staleness_watchdog(self):
"""Detect silent connection drop"""
while True:
await asyncio.sleep(10)
if self.last_message_at:
elapsed = (datetime.utcnow() - self.last_message_at).seconds
if elapsed > self.stale_threshold:
raise RuntimeError(f"Connection stale: {elapsed}s without data")
Exponential backoff reconnection and a staleness watchdog are the minimum for industrial-grade scraping.
How to manage an order book via WebSocket?
Most exchanges send order book updates as incremental changes—only levels that changed. Maintaining a local order book:
from sortedcontainers import SortedDict
class LocalOrderBook:
def __init__(self):
self.bids = SortedDict(lambda k: -k) # descending
self.asks = SortedDict() # ascending
self.last_update_id = 0
def apply_snapshot(self, snapshot: dict):
self.bids.clear()
self.asks.clear()
for price, qty in snapshot['bids']:
self.bids[float(price)] = float(qty)
for price, qty in snapshot['asks']:
self.asks[float(price)] = float(qty)
self.last_update_id = snapshot['lastUpdateId']
def apply_update(self, update: dict):
if update['u'] <= self.last_update_id:
return # stale update, ignore
for price, qty in update['b']: # bids
p, q = float(price), float(qty)
if q == 0:
self.bids.pop(p, None)
else:
self.bids[p] = q
for price, qty in update['a']: # asks
p, q = float(price), float(qty)
if q == 0:
self.asks.pop(p, None)
else:
self.asks[p] = q
self.last_update_id = update['u']
def best_bid(self) -> tuple[float, float]:
k = next(iter(self.bids))
return k, self.bids[k]
def best_ask(self) -> tuple[float, float]:
k = next(iter(self.asks))
return k, self.asks[k]
Important: on startup, get a snapshot via REST, then apply WebSocket updates starting with lastUpdateId > snapshotId. Updates before the snapshot are discarded; a gap in the U → u sequence requires a new snapshot.
Scaling: multiple pairs and exchanges
A single async event loop in Python handles 50–200 simultaneous WebSocket connections. For more, use multiple processes or a Go service (goroutines are significantly lighter than asyncio tasks).
Fanout results: processed messages are published to Redis Pub/Sub or Kafka for downstream consumers. The WebSocket handler should do minimal processing and publish quickly—heavy processing is done by a separate consumer.
Monitoring health
Metrics for each WebSocket connection: messages per second, reconnect count, last message timestamp, lag from exchange timestamp to processing timestamp. Use Grafana + Prometheus alerting on stale connections (no messages for active pair > 60 seconds).
| Metric | Description | Alert Threshold |
|---|---|---|
| messages/sec | Number of messages per second | < 0.5x expected |
| reconnects | Number of reconnections per hour | > 5 |
| last_message_age | Time since last message | > 60 s |
| lag | Delay from exchange time | > 500 ms |
What's included in WebSocket scraping setup
- Connection to exchanges / blockchain nodes via WebSocket (Binance, Coinbase, Kraken, Ethereum, Polygon, Solana, and others)
- Implementation of reconnect logic with exponential backoff and staleness watchdog
- Local order book aggregation with snapshot synchronization
- Publishing normalized data to Redis Pub/Sub or Kafka
- Monitoring and alerts (Grafana, Prometheus)
- Documentation of architecture and configuration
- Training your team on system operation
Our experience and guarantees
Over 5 years, we've completed 50+ real-time scraping projects for crypto exchanges, DeFi protocols, and NFT marketplaces. We guarantee stable operation, automatic recovery after failures, and 24/7 monitoring. We work with Ethereum, Binance, Polygon, Arbitrum, Solana, and other networks.
Setup of real-time scraping for 3–5 exchanges with monitoring of 20–50 pairs, reconnect logic, and publishing to Redis/Kafka takes 1–2 days. Contact us for a cost estimate. Order now to get a consultation.