Real-Time Order Book Data Scraping
Imagine your trading bot executes a trade at a price that has already changed – the order book is out of sync due to a WebSocket disconnect. Losses from a single such event can reach 2-3% of capital. We encountered this in early projects and developed a system that eliminates such incidents.
Order book data is critical for three scenarios: building a trading bot, creating a liquidity aggregator, and market monitoring. In each, the common challenge is stable high-frequency data acquisition without loss and with minimal latency. Wrong protocol choice or lack of reconnect handling leads to order book desync and unprofitable trades. Over the years, we have implemented over 30 integrations with various exchanges. We guarantee stable data collection and low latency.
Why WebSocket over REST?
REST polling (GET /api/v3/depth?symbol=BTCUSDT) is the wrong choice for real-time order book. On active markets, the order book updates 10–100 times per second. Polling once per second gives stale data and strains API rate limits. The right approach is WebSocket streams with incremental updates.
| Parameter | REST Polling | WebSocket Stream |
|---|---|---|
| Latency | 1 sec+ (poll interval) | 10-100 ms (event-driven) |
| API Load | High (requests every second) | Low (single connection) |
| Data Freshness | Instantly outdated | Always latest state |
| Scaling | Issues with multiple instruments | Up to 1024 streams per key |
Most major CEXes (Binance, Bybit, OKX) follow the same scheme:
- Obtain a snapshot via REST (full order book at current moment)
- Subscribe to a WebSocket update stream
- Apply updates to the snapshot, maintaining a local copy of the order book
import asyncio, json, aiohttp from sortedcontainers import SortedDict class OrderBook: def __init__(self): self.bids = SortedDict(lambda x: -x) # descending order self.asks = SortedDict() self.last_update_id = 0 def apply_update(self, bids: list, asks: list, update_id: int): if update_id <= self.last_update_id: return # stale update, ignore for price, qty in bids: price, qty = float(price), float(qty) if qty == 0: self.bids.pop(price, None) # remove level else: self.bids[price] = qty for price, qty in asks: price, qty = float(price), float(qty) if qty == 0: self.asks.pop(price, None) else: self.asks[price] = qty self.last_update_id = update_id @property def best_bid(self) -> tuple[float, float] | None: if self.bids: price = self.bids.keys()[0] return price, self.bids[price] return None @property def best_ask(self) -> tuple[float, float] | None: if self.asks: price = self.asks.keys()[0] return price, self.asks[price] return None How to Resynchronize the Order Book After a Disconnect?
When a connection drops or packets are lost, the risk of inconsistency is high. We use the following techniques:
- Buffer updates until a snapshot is received (as shown above)
- Check the sequence ID of each update: if
update_iddoes not match the expected one, discard the packet and request a new snapshot - Exponential backoff on reconnection with a cap of 60 seconds
- Monitor latency and alert when threshold is exceeded (e.g., >500 ms)
async def connect_ws_with_retry(url: str, handler, max_retries=10): for attempt in range(max_retries): try: async with websockets.connect(url, ping_interval=20) as ws: async for message in ws: await handler(message) except (websockets.exceptions.ConnectionClosed, Exception) as e: wait = min(2 ** attempt, 60) # max 60 seconds logging.warning(f"WS disconnected: {e}, retry in {wait}s") await asyncio.sleep(wait) Details of Working with Binance Depth Stream
Binance is the most frequent request. They have two stream variants:
-
btcusdt@depth— updates every 100ms or 1000ms (parameter@depth@100ms) -
btcusdt@depth20— top-20 levels every 100ms (no incremental updates, always full)
For the full order book with patching:
async def maintain_binance_orderbook(symbol: str): ob = OrderBook() buffer = [] # buffer updates until snapshot received async def handle_ws_message(msg): data = json.loads(msg) # Accumulate updates UNTIL we get snapshot if ob.last_update_id == 0: buffer.append(data) return # Binance: update valid if U <= lastUpdateId+1 <= u if data['U'] <= ob.last_update_id + 1 <= data['u']: ob.apply_update(data['b'], data['a'], data['u']) # Start WS ws_task = asyncio.create_task(connect_ws( f"wss://stream.binance.com:9443/ws/{symbol.lower()}@depth@100ms", handle_ws_message )) # Get snapshot (wait a bit for buffer to accumulate) await asyncio.sleep(0.5) async with aiohttp.ClientSession() as session: async with session.get( f"https://api.binance.com/api/v3/depth", params={"symbol": symbol.upper(), "limit": 1000} ) as resp: snapshot = await resp.json() # Initialize order book from snapshot for price, qty in snapshot['bids']: ob.bids[float(price)] = float(qty) for price, qty in snapshot['asks']: ob.asks[float(price)] = float(qty) ob.last_update_id = snapshot['lastUpdateId'] # Apply buffered updates for update in buffer: if update['u'] > ob.last_update_id: ob.apply_update(update['b'], update['a'], update['u']) await ws_task Critical point: if an update is missed (gap in U → u sequence) the order book becomes desynchronized. Resync logic is needed: detect the gap and reinitialize from a new snapshot.
Case Study: Aggregating Binance and Bybit for Arbitrage
For cross-exchange arbitrage, it is necessary to maintain order books of multiple exchanges in parallel. Here is an example of an aggregator that finds the best price:
EXCHANGES = { "binance": BinanceOrderBook, "bybit": BybitOrderBook, "okx": OKXOrderBook, } async def run_aggregator(symbol: str): books = {name: cls(symbol) for name, cls in EXCHANGES.items()} tasks = [book.run() for book in books.values()] await asyncio.gather(*tasks) def get_best_price_across_exchanges(books: dict[str, OrderBook]) -> dict: best_bids = [(name, *ob.best_bid) for name, ob in books.items() if ob.best_bid] best_asks = [(name, *ob.best_ask) for name, ob in books.items() if ob.best_ask] best_bids.sort(key=lambda x: x[1], reverse=True) best_asks.sort(key=lambda x: x[1]) return { "best_bid": {"exchange": best_bids[0][0], "price": best_bids[0][1], "qty": best_bids[0][2]}, "best_ask": {"exchange": best_asks[0][0], "price": best_asks[0][1], "qty": best_asks[0][2]}, "spread": best_asks[0][1] - best_bids[0][1] } Real-time aggregation allows seeing the best bid and ask across all exchanges. This is the foundation for arbitrage strategies and building a unified order book.
Data Storage: TimescaleDB vs File-Based
For backtesting or auditing, it is better to store the stream of updates rather than only snapshots. L2 order book updates generate a large volume: for BTC/USDT on Binance ~100MB/hour of uncompressed data.
| Criterion | TimescaleDB | File-Based (Parquet) |
|---|---|---|
| Real-time queries | Yes (SQL) | No (analytics only) |
| Compression | Automatic | Configurable (lz4) |
| Stream replay | Requires additional processing | Direct read |
| Kafka integration | Yes | Not available |
We recommend using TimescaleDB for long-term storage and Parquet for analytics. If needed, we integrate the stream into Kafka for downstream systems.
# Writing to binary format via msgpack import msgpack, lz4.frame def serialize_update(update: dict) -> bytes: packed = msgpack.packb(update, use_bin_type=True) return lz4.frame.compress(packed) # TimescaleDB for time-series storage # Hypertable automatically partitions by time CREATE TABLE ob_updates ( time TIMESTAMPTZ NOT NULL, exchange TEXT NOT NULL, symbol TEXT NOT NULL, side CHAR(1) NOT NULL, -- 'b' or 'a' price NUMERIC NOT NULL, quantity NUMERIC NOT NULL ); SELECT create_hypertable('ob_updates', 'time'); What’s Included in the Work?
When ordering an order book scraping system, you get:
- An architectural solution with protocol selection and resynchronization strategy
- Source code in Python with asyncio and deployment documentation
- Grafana dashboards for monitoring latency and errors
- 99.9% stability guarantee and two weeks of support after deployment
Development timeline: 2 to 4 weeks depending on the number of exchanges and required architecture. Cost is calculated individually. Contact us to discuss your project – we will find an optimal solution.
Common Mistakes in Order Book Scraping
- Ignoring sequence ID and lack of resynchronization after a gap
- Using REST polling instead of WebSocket (leads to delays and rate limits)
- Wrong order: snapshot first, then subscription to updates
- No buffering of updates before snapshot (first packets are lost)
- Incorrect reconnect handling without exponential backoff
Order the development of an order book scraping system with a guarantee of stability and low latency. Our experience in high-frequency trading enables us to create solutions that do not lose data and do not desync even under peak loads.