A full order book contains all liquidity information on an exchange. Collecting it, normalizing it, and turning it into features for machine learning is a non-trivial engineering challenge. We have built a production-grade pipeline for Binance, Bybit, and OKX that processes up to 10,000 updates per second. Our experience includes integration with 15+ crypto exchanges and storing about 5 TB of data per month. A full L2 order book describes every price level with volume — this is the foundation for building short-term predictions. Stable collection under peak loads and snapshot consistency is guaranteed.
Customers often arrive with raw WebSocket streams, unsure how to synchronize the diff stream with a REST snapshot. An off-by-one error causes the book to diverge, leading to incorrect signals. We solve this at the collector architecture level.
Problems we solve
- Data volume. A full L2 order book on Binance contains 5000 levels on each side. With updates every 100 ms, this generates tens of gigabytes per day. Naive storage in PostgreSQL will kill performance.
- Race conditions. The WebSocket diff stream arrives asynchronously. Without synchronization with the REST snapshot, the book diverges — prices go to non-existent levels.
- Data format. Each exchange delivers the order book differently: Binance uses nested arrays, Coinbase uses JSON with different keys. A unified interface is needed.
How to synchronize WebSocket diff stream with REST snapshot?
The algorithm is simple: open a WebSocket, get the first diff stream, immediately request a full REST snapshot. Then apply each update to the local order book. Use lastUpdateId for control: apply only messages with u > lastUpdateId. If the sequence is broken — re-request the snapshot. This approach eliminates book divergence even under high volatility.
How to collect order book via WebSocket: step-by-step algorithm
- Establish connection: via
wss://stream.binance.com:9443/ws/btcusdt@depth@100ms(analogue for other exchanges). - Initial REST snapshot: synchronize via
updateIdto ensure consistency. - Incremental updates: each diff stream message is applied to the current book state.
- Save snapshots: at a given periodicity (every N-th update), fix the full state for subsequent feature engineering.
Example collector code:
import asyncio
import websockets
import json
from collections import deque
class OrderBookCollector:
def __init__(self, symbol, max_depth=100):
self.symbol = symbol
self.bids = {}
self.asks = {}
self.max_depth = max_depth
self.snapshots = deque(maxlen=10000)
async def connect_binance(self):
url = f"wss://stream.binance.com:9443/ws/{self.symbol.lower()}@depth@100ms"
async with websockets.connect(url) as ws:
await self.fetch_snapshot()
async for msg in ws:
data = json.loads(msg)
self.process_diff_update(data)
if len(self.snapshots) % 10 == 0:
self.save_snapshot()
def process_diff_update(self, data):
for bid_level in data.get('b', []):
price, qty = float(bid_level[0]), float(bid_level[1])
if qty == 0:
self.bids.pop(price, None)
else:
self.bids[price] = qty
for ask_level in data.get('a', []):
price, qty = float(ask_level[0]), float(ask_level[1])
if qty == 0:
self.asks.pop(price, None)
else:
self.asks[price] = qty
def get_features(self, n_levels=20):
sorted_bids = sorted(self.bids.items(), reverse=True)[:n_levels]
sorted_asks = sorted(self.asks.items())[:n_levels]
if not sorted_bids or not sorted_asks:
return None
mid_price = (sorted_bids[0][0] + sorted_asks[0][0]) / 2
features = {}
for i, (price, qty) in enumerate(sorted_bids[:10]):
features[f'bid_qty_{i}'] = qty
features[f'bid_dist_{i}'] = (mid_price - price) / mid_price
for i, (price, qty) in enumerate(sorted_asks[:10]):
features[f'ask_qty_{i}'] = qty
features[f'ask_dist_{i}'] = (price - mid_price) / mid_price
bid_vol_n = sum(qty for _, qty in sorted_bids[:5])
ask_vol_n = sum(qty for _, qty in sorted_asks[:5])
features['obi_5'] = (bid_vol_n - ask_vol_n) / (bid_vol_n + ask_vol_n + 1e-8)
bid_vol_20 = sum(qty for _, qty in sorted_bids[:20])
ask_vol_20 = sum(qty for _, qty in sorted_asks[:20])
features['obi_20'] = (bid_vol_20 - ask_vol_20) / (bid_vol_20 + ask_vol_20 + 1e-8)
features['wmid'] = (sorted_bids[0][0] * sorted_asks[0][1] + sorted_asks[0][0] * sorted_bids[0][1]) / (sorted_bids[0][1] + sorted_asks[0][1])
features['spread'] = (sorted_asks[0][0] - sorted_bids[0][0]) / mid_price
for n in [5, 10, 20]:
bid_depth = sum(qty for _, qty in sorted_bids[:n])
ask_depth = sum(qty for _, qty in sorted_asks[:n])
features[f'depth_ratio_{n}'] = bid_depth / max(ask_depth, 1e-8)
return features
Why ClickHouse is optimal storage for order book?
Full L2 order book is huge. ClickHouse is 10x faster than PostgreSQL on columnar aggregations. According to ClickHouse documentation, a columnar DBMS provides up to 10x compression and write speeds over 1 million rows per second. Compare:
| DBMS | Write speed (rows/s) | Compression | Time-based aggregations |
|---|---|---|---|
| PostgreSQL | ~100,000 | 2-5x | Slow |
| TimescaleDB | ~200,000 | 3-6x | Medium |
| ClickHouse | ~1,000,000 | 5-10x | Fast |
Example schema with automatic TTL:
CREATE TABLE order_book_snapshots (
timestamp DateTime64(3),
symbol LowCardinality(String),
exchange LowCardinality(String),
bid_price_0 Float32, bid_qty_0 Float32,
bid_price_1 Float32, bid_qty_1 Float32,
-- ... up to bid_price_19, bid_qty_19
ask_price_0 Float32, ask_qty_0 Float32,
-- ...
spread Float32,
obi_5 Float32,
obi_20 Float32
) ENGINE = MergeTree()
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY (symbol, timestamp)
TTL timestamp + INTERVAL 90 DAY;
Infrastructure savings when using ClickHouse reach 70% due to compression — that's about $20,000 per year for a project with 5 TB of data. For large projects, savings can be up to $30,000 per year.
Feature engineering from order book
Based on collected snapshots, we build features. Basic: OBI (order book imbalance), spread, depth. Additional: moving averages of OBI, its volatility, cumulative order flow (COF).
def engineer_orderbook_features(snapshots_df, window_sizes=[10, 50, 100]):
features = snapshots_df.copy()
for window in window_sizes:
features[f'obi_5_ma_{window}'] = features['obi_5'].rolling(window).mean()
features[f'obi_5_delta_{window}'] = features['obi_5'].diff(window)
features[f'obi_5_std_{window}'] = features['obi_5'].rolling(window).std()
features['cof'] = features['obi_5'].cumsum()
features['cof_ma'] = features['cof'].rolling(100).mean()
features['cof_deviation'] = features['cof'] - features['cof_ma']
features['spread_ma'] = features['spread'].rolling(50).mean()
features['spread_ratio'] = features['spread'] / features['spread_ma']
features['depth_change'] = features['depth_ratio_10'].diff(10)
return features
How to evaluate mid-price prediction quality?
For short-term mid-price prediction (after N book updates) we use accuracy, precision, and F1-score for binary classification of direction. Code for training data preparation:
def create_training_data(snapshots_df, prediction_horizon=10):
features = engineer_orderbook_features(snapshots_df)
future_mid = snapshots_df['mid_price'].shift(-prediction_horizon)
current_mid = snapshots_df['mid_price']
target = np.sign(future_mid - current_mid)
valid_mask = features.notna().all(axis=1) & target.notna()
return features[valid_mask], target[valid_mask]
Common mistakes in order book pipeline development
Even experienced teams make mistakes: ignoring book skew during high volatility, incorrect handling of lastUpdateId events, lack of consistency checks after reconnect. We encountered a project where due to missed diffs the book diverged by 20% — the model gave false signals. The solution is embedding checksum checks and automatic full snapshot recovery upon detecting inconsistency.
What's included in pipeline development
- Source code for the collector and pipeline (async Python).
- Test data dumps for offline testing.
- README with detailed usage examples.
- ClickHouse schema migrations with TTL.
- Training your team on pipeline usage.
Stages and timelines
| Stage | Duration | Result |
|---|---|---|
| Analytics | 2-3 days | API specification, volume estimates |
| Design | 2-3 days | Storage schema, feature selection |
| Implementation | 5-10 days | Collector, pipeline, code |
| Testing | 3-5 days | 24h simulation, reports |
| Deployment | 2-3 days | Docker, monitoring |
A basic pipeline for one exchange with a LightGBM model takes from 14 to 30 working days. Development costs start at $15,000. We provide an exact estimate after a free audit of your data. Request an analysis and we'll pick the optimal architecture for your order book volume. With over 5 years of experience and 50+ completed projects, we ensure reliable delivery. Contact us for a consultation.