Tick-Data Pipeline Development for ML

We develop tick-data processing pipelines — recording every trade with price, volume, and side. Standard OHLCV candles lose market microstructure: liquidity imbalance, large trades, buy/sell flow. Without a quality pipeline, an ML model trains on noise. For example, in one project for Binance (from our practice), the load reached 300,000 ticks per second — ClickHouse handled it, while PostgreSQL crashed at 10,000. Our over five years of experience guarantees reliability. Contact us — we are ready to design and implement a pipeline for your tasks.

Why Tick Data Matters More Than OHLCV for ML

When aggregating into 1-minute candles, up to 80% of information is lost: you do not see how trades are distributed within the interval, whether there was a volume spike, or who was the aggressor. Volume bars, dollar bars, and imbalance bars preserve these signals. ML models trained on ticks show 15–20% higher accuracy in price direction prediction tasks.

Problems We Solve

• High loads. Exchanges generate up to 500,000 ticks per second. Standard databases cannot handle such insertion rates.
• Latency. For HFT strategies, the delay from tick receipt to signal must not exceed 10 ms.
• Storage. Tick data for a year amounts to tens of terabytes. Partitioning, TTL, and efficient compression are necessary. ClickHouse infrastructure savings can reach 50% compared to traditional relational databases.
• Bar diversity. Time bars are uneven during low activity periods. Volume/dollar/imbalance bars adapt to market activity.

How We Do It: Stack and Case Study

In one project for Binance (from our practice), we built a pipeline that collects aggregated trades via WebSocket, buffers them in memory, and asynchronously inserts into ClickHouse.

import asyncio import websockets import json from datetime import datetime import asyncpg class TickDataCollector: def __init__(self, symbol, db_pool): self.symbol = symbol self.db_pool = db_pool self.buffer = [] self.buffer_size = 1000 async def connect_binance_trades(self): url = f"wss://stream.binance.com:9443/ws/{self.symbol.lower()}@aggTrade" async with websockets.connect(url, ping_interval=20) as ws: async for msg in ws: trade = json.loads(msg) tick = { 'symbol': self.symbol, 'timestamp': datetime.fromtimestamp(trade['T'] / 1000), 'price': float(trade['p']), 'quantity': float(trade['q']), 'is_buyer_maker': trade['m'], 'trade_id': trade['a'] } self.buffer.append(tick) if len(self.buffer) >= self.buffer_size: await self.flush_to_db() async def flush_to_db(self): async with self.db_pool.acquire() as conn: await conn.executemany( """INSERT INTO trades (symbol, timestamp, price, quantity, is_buyer_maker, trade_id) VALUES ($1, $2, $3, $4, $5, $6)""", [(t['symbol'], t['timestamp'], t['price'], t['quantity'], t['is_buyer_maker'], t['trade_id']) for t in self.buffer] ) self.buffer.clear() 

Storage is organized in ClickHouse with the MergeTree engine, daily partitioning, and a TTL of 365 days. This provides efficient compression (10x compared to CSV) and high insertion speed.

CREATE TABLE trades ( timestamp DateTime64(3), symbol LowCardinality(String), price Float64, quantity Float32, is_buyer_maker UInt8, trade_id UInt64 ) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(timestamp) ORDER BY (symbol, timestamp) TTL timestamp + INTERVAL 365 DAY SETTINGS index_granularity = 8192; 

ClickHouse inserts 500K+ rows/sec — 50 times faster than PostgreSQL for such loads. Monthly aggregations complete in seconds. We guarantee your pipeline will handle any market activity.

How to Build Volume Bars from Ticks: Step by Step

1. Connect to the exchange WebSocket to receive aggregated trades.
2. Accumulate ticks in a buffer (e.g., 1000 records).
3. When the specified volume is reached, close the bar and save it to ClickHouse.
4. Use the create_volume_bars function from the example below.

Volume bars close when a given volume accumulates, not at a fixed time interval. This yields a uniform number of observations regardless of market activity.

def create_volume_bars(ticks_df, bar_volume=10): """Each bar = bar_volume units of the asset""" bars = [] current_bar = {'open': None, 'high': -np.inf, 'low': np.inf, 'close': None, 'volume': 0, 'start_time': None} for _, tick in ticks_df.iterrows(): if current_bar['open'] is None: current_bar['open'] = tick['price'] current_bar['start_time'] = tick['timestamp'] current_bar['high'] = max(current_bar['high'], tick['price']) current_bar['low'] = min(current_bar['low'], tick['price']) current_bar['close'] = tick['price'] current_bar['volume'] += tick['quantity'] if current_bar['volume'] >= bar_volume: bars.append(current_bar.copy()) current_bar = {'open': None, 'high': -np.inf, 'low': np.inf, 'close': None, 'volume': 0, 'start_time': None} return pd.DataFrame(bars) 

Similarly, dollar bars (by USD volume) and imbalance bars (by buy/sell imbalance) are constructed.

What Benefits Does Tick Feature Engineering Provide?

Features extracted from ticks improve ML model quality: flow imbalance, trade frequency, VWAP deviation, large trade ratio. In a real-time streaming ML pipeline, these features are calculated on sliding windows.

def create_tick_features(ticks_df, window_ticks=[50, 200, 1000]): features = [] for i in range(max(window_ticks), len(ticks_df)): row_features = {} for window in window_ticks: window_data = ticks_df.iloc[i-window:i] buy_vol = window_data[~window_data['is_buyer_maker']]['quantity'].sum() sell_vol = window_data[window_data['is_buyer_maker']]['quantity'].sum() row_features[f'flow_imbalance_{window}'] = ( (buy_vol - sell_vol) / (buy_vol + sell_vol + 1e-8) ) row_features[f'trade_frequency_{window}'] = ( window / (window_data['timestamp'].max() - window_data['timestamp'].min()).total_seconds() + 1e-8 ) row_features[f'avg_trade_size_{window}'] = window_data['quantity'].mean() row_features[f'large_trade_ratio_{window}'] = ( (window_data['quantity'] > window_data['quantity'].quantile(0.9)).mean() ) vwap = (window_data['price'] * window_data['quantity']).sum() / window_data['quantity'].sum() row_features[f'vwap_deviation_{window}'] = ( ticks_df.iloc[i]['price'] - vwap ) / vwap features.append(row_features) return pd.DataFrame(features) 

Large trades (above the 99th percentile) often indicate institutional activity. Analyzing their direction provides an additional signal.

How to Ensure Latency <10 ms?

Real streaming architecture:

Binance WebSocket → asyncio consumer → buffer → ClickHouse batch insert → Redis sorted set (last 10k ticks) → Feature calculator (sliding window) → ML inference → Signal output 

Latency from tick to signal is under 10 ms. Achieved through asynchronous I/O, Redis buffering, and precomputed features on time windows. Savings on ClickHouse cluster compared to traditional databases can reach 50%.

According to ClickHouse documentation, insertion speed reaches 500,000 rows per second ClickHouse Documentation.

Process of Work

Timeline and Deliverables

Base version (single symbol, ClickHouse, Redis) — from 2 weeks. Full pipeline with volume/dollar/imbalance bars, feature engineering, and real-time inference — from 4 weeks. Project cost varies; exact pricing is determined after analysis. Investment in a quality pipeline pays off through improved ML model accuracy for trading.

What is included:

• Architecture documentation.
• Pipeline source code with comments.
• ClickHouse, Redis, and queue setup.
• Integration with your ML infrastructure.
• Team training (2–3 calls).
• 2 months of support after deployment.

Common Mistakes

• Too fine partitioning (by hour): large number of partitions degrades ClickHouse performance. Optimal is by day.
• Ignoring TTL: without automatic data cleanup, the disk fills up within a month.
• Using time bars for low-liquidity assets: most candles will be empty.

We have been developing turnkey tick-data pipelines for over five years, implementing 30+ projects for crypto trading. Get a free analysis of your data and recommendations for pipeline optimization. Contact us — we will evaluate your project and propose the best solution.