Real Task: Predicting Price Movement from the Order Book

A trader looks at the Depth of Market (DOM) and sees an imbalance: 2000 lots at Best Bid, 1500 at Best Ask. The order book is the full snapshot of limit orders with prices and volumes. In real time, it provides a signal milliseconds before trade execution. Extracting a reliable trading signal from noise is key. We tackle the problem: a trader needs to predict short-term price movement based on L2 data. We demonstrate using a project for Binance: transforming the order flow into predictions with a 1-10 second horizon. We combine linear models and convolutional neural networks for maximum accuracy at minimal latency.

Why Order Book Analysis Works for Short-Term Forecasting

Market microstructure is encoded in the distribution of liquidity. The order book reflects participants' expectations – the imbalance between volumes at the best levels precedes price movement. Order Book Imbalance (OBI) is the simplest yet powerful predictor. For 1–10 second horizons, OBI achieves AUC 0.58 with 0.1 ms latency. That suffices for arbitrage strategies. For trend trading, a more complex model capturing book dynamics is needed.

Order Book Data Structure

L2 Order Book snapshot:

Price    | Bid Volume | Ask Volume
---------|-----------|----------
100.05   |     0     |   5000
100.04   |     0     |   3000
100.03   |     0     |   1500  ← Best Ask
100.02   |   2000     |     0   ← Best Bid
100.01   |   3500     |     0
100.00   |   8000     |     0
99.99    |   2500     |     0

L3 order book contains individual orders with IDs – needed for microstructure analysis, available on some exchanges (Binance, CME via API).

Feature Engineering from Order Book

Basic metrics: bid-ask spread, mid price, imbalance, weighted mid price.

Order Book Imbalance (OBI):

def order_book_imbalance(book, levels=5):
    bids = [vol for price, vol in book['bids'][:levels]]
    asks = [vol for price, vol in book['asks'][:levels]]
    return (sum(bids) - sum(asks)) / (sum(bids) + sum(asks))

OBI > 0 → buyer pressure → expected upward move. This is one of the strongest short-term predictors (1-10 second horizon).

Iceberg detection: hidden orders – series of small orders at the same price. Signs: quick replenishment of a level after execution, constant volume at a level.

Market depth curves:

def depth_imbalance_at_level(book, price_distance):
    bid_vol = sum([vol for p, vol in book['bids'] if (mid - p) <= price_distance])
    ask_vol = sum([vol for p, vol in book['asks'] if (p - mid) <= price_distance])
    return (bid_vol - ask_vol) / (bid_vol + ask_vol)

Features: imbalance at 0.1%, 0.3%, 0.5%, 1.0% from mid. We also add the rate of change of imbalance and the slope of the depth curve.

Choosing Between OBI and DeepLOB

Model comparison:

OBI is a linear predictor with 0.1 ms inference latency – critical for HFT. LightGBM requires 0.5 ms, DeepLOB 2 ms. But DeepLOB gives a 5% AUC boost over gradient boosting, justified for large capital strategies. We choose the approach per task: for arbitrage, OBI suffices; for trend trading, DeepLOB.

How DeepLOB Processes L2 Data

The convolutional neural network takes time slices of the order book as a 2D image (price × level × volume). It trains on millions of examples. In production we use ONNX Runtime for inference – p99 latency under 5 ms on a T4 GPU.

Iceberg Orders and Their Detection

Iceberg orders mask true volume. We use statistics of order reappearance at the same level: if the volume recovers more than 3 times in 1 second, it's likely an iceberg. This feature improves prediction accuracy by 2-3%.

How We Build a Production Solution

Our team has over 7 years in algorithmic trading. We use: PyTorch, ONNX Runtime for inference, Kafka for data streams.

Process

1. Data collection: obtain historical L2 data via Binance WebSocket or CME FIX.
2. Feature engineering: compute OBI, depth curves, imbalance at multiple horizons.
3. Model design: try DeepLOB, LightGBM on handcrafted features, hybrid approaches.
4. Training: on a GPU cluster with distributed training.
5. Backtesting: on historical data accounting for fees and slippage.
6. Deployment: containerization, ONNX Runtime on inference server, integration with OMS.

What’s Included

• Model card with metrics and limitations.
• API for signal retrieval (gRPC or REST).
• Training of client team on interpreting outputs.
• Support for 3 months post-deployment.
• Source code and pipeline configuration.

We guarantee transparency – you get an interpretable pipeline, not a black box. All results are confirmed by backtests on a hold-out set. Cost savings from ready-made feature extractors and base models reach 30% of budget.

Common Mistakes in Building Order Book Models

• Ignoring microstructure: using only mid price without depth.
• Incorrect forecast horizon: for HFT milliseconds matter, for trends seconds.
• Lack of iceberg detection: hidden orders skew imbalance.

Development Timelines

Pricing is individual – depends on data volume, model complexity, and infrastructure. Contact us for a preliminary assessment of your task – we will select the optimal architecture for your strategy. Get a consultation for your project right now.

When does a time series forecasting model fail in production?

The CFO requests a quarterly sales forecast. An analyst builds SARIMA on three years of data, achieves MAPE 8.3% on the test set, and deploys. Two months later, the metric in production jumps to 23%. The root cause: the model was trained on pre‑COVID data, tested on a stable period, but production hit a promotion and supply chain disruption. Data leakage plus distribution shift—perfect notebook numbers, a broken forecast in reality. We have seen this pattern dozens of times across retail, fintech, and IoT. Our team has delivered more than 50 forecasting projects over 5+ years.

Incorrect cross-validation. Standard train_test_split for time series creates data leakage: the model sees future values during training. The correct approach is TimeSeriesSplit or walk‑forward validation with an expanding window.

Multiple seasonality. Hourly electricity consumption has three seasonalities: daily (24h), weekly (168h), yearly (8760h). SARIMA handles only one. Prophet can handle multiple but scales poorly to thousands of series.

Missing values and anomalies. A missing sensor reading is information (the sensor turned off), not NaN. Linear interpolation destroys this signal. Proper handling depends on the missingness mechanism.

Cold start. A new SKU in a 50,000‑item assortment has no history, yet a forecast is needed. Standard approaches fail; cross‑learning or feature‑based methods are required.

Why is model selection critical for your data?

Prophet (Meta) – a solid start for business data with clear seasonality and holidays. Fast setup, interpretable, built‑in outlier detection. Fails on irregular patterns and does not scale beyond ~10k series without parallelization.

Gradient boosting on features (LightGBM, XGBoost) – often underestimated. Engineer lags (t‑1, t‑7, t‑28), rolling means, day‑of‑week, holidays. The model trains on all series simultaneously, solving cold start via transfer learning. MAPE in retail often beats neural nets with proper feature engineering.

TFT (Temporal Fusion Transformer) – a transformer designed for interpretable forecasting with covariates. Built‑in variable selection, temporal attention, quantile outputs. Available in pytorch‑forecasting. Requires ~10,000+ records per series for stable training.

PatchTST – splits the series into patches (like ViT for images), capturing local patterns better than classic transformers. Excellent for long‑horizon forecasting (96–720 steps ahead).

N‑HiTS, N‑BEATS – attention‑free neural architectures, faster than TFT, competitive accuracy. N‑BEATS won the M4/M5 benchmarks for tasks without covariates.

How do we deploy TFT in production?

A typical pipeline via pytorch‑forecasting:

training = TimeSeriesDataSet(
    data,
    time_idx="time_idx",
    target="sales",
    group_ids=["store", "sku"],
    min_encoder_length=max_encoder_length // 2,
    max_encoder_length=max_encoder_length,  # 120 days
    min_prediction_length=1,
    max_prediction_length=max_prediction_length,  # 28 days
    static_categoricals=["store_type", "category"],
    time_varying_known_reals=["price", "promo_flag"],
    time_varying_unknown_reals=["sales"],
    target_normalizer=GroupNormalizer(groups=["store", "sku"], transformation="softplus"),
)

A common mistake: the default target_normalizer (StandardScaler) breaks predictions for series with zero values (no sales on weekends). GroupNormalizer with transformation="softplus" is the correct choice for count data.

Case study: retail demand forecasting

A chain of 120 stores, 8,000 SKUs, 28‑day forecast horizon. The original system: SARIMA per series, MAPE 18.4%, retraining cycle – 6 hours. We replaced it with TFT on PyTorch + pytorch‑forecasting: a single model for all series, MAPE 11.2%, retraining – 40 minutes on an A10G. Feature importance via variable selection revealed that day_before_holiday influences more than the holiday date itself. Annual savings on inference alone exceeded $50,000.

Step‑by‑step configuration

1. Data collection and preparation. Handle missing values (mark NaN, interpolate only for technical failures), aggregate to required frequency, engineer covariates (holidays, promotions, prices).
2. Create TimeSeriesDataSet. Set group_ids (store + SKU), time index, forecast horizon. Choose target_normalizer based on target distribution.
3. Train a baseline. Prophet or LightGBM first – to understand complexity.
4. Train TFT. Use TemporalFusionTransformer with loss=QuantileLoss(), tune learning rate and hidden layer sizes.
5. Validate and interpret. Walk‑forward test, analyze variable selection, build attention heatmaps.

How to properly evaluate forecast quality?

RMSE alone is misleading – it over‑penalizes large values. Our standard set:

• MAPE – interpretable, unstable near zero.
• sMAPE – symmetric, avoids division by small numbers.
• MASE (Mean Absolute Scaled Error) – normalized relative to a naive seasonal forecast, ideal for comparing series of different scales.
• Pinball loss – for probabilistic forecasting, inventory management.

We guarantee a model card with these metrics on the validation set and walk‑forward results on at least 6 months of history.

What deliverables do you receive?

• Documentation of chosen architecture and hyperparameter rationale.
• Reproducible training and inference pipeline (Docker + CI/CD + Airflow/Prefect).
• Committed code with unit tests for key components.
• Team training: retraining, output interpretation, deployment of new versions.
• 3 months of post‑delivery support (consultations, bug fixes, fine‑tuning).

The model is deployed via FastAPI or Triton Inference Server. Retraining is scheduled (e.g., weekly) via Airflow with drift validation and automatic rollback if metrics deteriorate.

Process and timeline

We start with EDA: visualization, ADF test, STL decomposition, analysis of missing values and outliers. This takes 2–3 days but often reveals systemic data issues that block forecasting. Then we build a baseline (naive seasonal, Prophet), engineer features for LightGBM, and select a neural architecture if needed. Walk‑forward validation with a realistic horizon. Deployment via API with automatic retraining scheduled via Airflow or Prefect.

Timeline: MVP forecast on one data type – 3–6 weeks. Hierarchical forecasting system with automation – 2–5 months. Cost is calculated individually based on data volume, number of series, and required accuracy.

Our team consists of certified ML engineers (AWS ML Specialty, GCP Professional ML Engineer) with 5+ years on the market and over 50 completed forecasting projects. Contact us for a free analysis of your data – we will assess the task and provide initial recommendations within 1–2 days. Request a consultation to ensure your forecasts work in production, not just in a notebook.