Exchange Data Normalization System Development

One of our clients, a hedge fund, managed a portfolio across 10 exchanges and spent three days a week manually reconciling disparate tickers. After implementing a normalization system, that time dropped to one hour. We've been building such systems for over a decade, integrating with 20+ exchanges — from Binance to decentralized protocols. Without normalization, you get scattered data that's useless for trading, analytics, or backtesting. We solve this by creating a unified data model that hides all exchange-specific nuances behind a single interface.

But the problem runs deeper than it seems. Even after normalizing symbols and timestamps, issues remain: how to handle API errors, validate data, and scale when adding new exchanges? In this article, we share concrete solutions we use in commercial projects.

What needs to be normalized

Symbols and pairs. Each exchange has its own conventions. Normalized format: BASE/QUOTE in uppercase — BTC/USDT, ETH/BTC. Exchange symbols are stored in a mapping with reverse conversion support.

Timestamps. Binance returns milliseconds, some exchanges return seconds, OKX returns nanoseconds. Normalized format: UTC milliseconds stored as int64.

Numbers. REST APIs often return numbers as strings ("43250.50"), some exchanges drop trailing zeros. Normalized format: Decimal with explicit precision depending on the instrument.

Order sides. BUY/SELL, buy/sell, b/s, 1/-1 — all exist. Normalized format: enum BUY | SELL. Order statuses. Each exchange has its own set. Normalized mapping:

How we approach normalization

We build the normalizer as a set of exchange-specific adapters sharing a common interface. This allows adding new exchanges without modifying existing code. We use async Python and pydantic for strict input schema validation.

from abc import ABC, abstractmethod from decimal import Decimal class ExchangeNormalizer(ABC): @abstractmethod def normalize_symbol(self, raw_symbol: str) -> str: """Convert exchange symbol to normalized BASE/QUOTE format""" @abstractmethod def normalize_ticker(self, raw_data: dict) -> NormalizedTicker: """Normalize ticker data""" @abstractmethod def normalize_order(self, raw_data: dict) -> NormalizedOrder: """Normalize order data""" class BinanceNormalizer(ExchangeNormalizer): SYMBOL_MAP = { "BTCUSDT": "BTC/USDT", "ETHUSDT": "ETH/USDT", # ... from /api/v3/exchangeInfo } def normalize_ticker(self, raw: dict) -> NormalizedTicker: return NormalizedTicker( exchange="binance", symbol=self.normalize_symbol(raw["s"]), timestamp=int(raw["T"]), price=Decimal(raw["c"]), volume_24h=Decimal(raw["v"]), ) 

Dynamic loading of symbol mapping

Hardcoding symbol mappings is a bad idea: exchanges add new pairs daily. The right approach is to load the mapping from the Exchange Info API at startup and update periodically:

async def load_symbol_map(self): exchange_info = await self.rest_client.get("/api/v3/exchangeInfo") self.symbol_map = { s["symbol"]: f"{s['baseAsset']}/{s['quoteAsset']}" for s in exchange_info["symbols"] if s["status"] == "TRADING" } # Reverse mapping for converting back self.reverse_map = {v: k for k, v in self.symbol_map.items()} 

We regularly check for updates via the Binance API documentation to keep the mapping current.

Validating normalized data

After normalization, it's crucial to validate the output. Negative prices, zero volumes, timestamps in the future — all are signs of source data issues:

def validate_ticker(ticker: NormalizedTicker) -> list[str]: errors = [] if ticker.price <= 0: errors.append(f"Invalid price: {ticker.price}") if ticker.timestamp > now_ms() + 5000: errors.append(f"Future timestamp: {ticker.timestamp}") if ticker.bid and ticker.ask and ticker.bid >= ticker.ask: errors.append(f"Crossed book: bid={ticker.bid} ask={ticker.ask}") return errors 

Invalid data is logged and discarded, never reaching downstream systems. This ensures your algorithms always receive correct data.

Why normalization is critical for your project

Poor normalization leads to incorrect backtest results, erroneous orders, and lost money. Our approach reduces data errors by 80% compared to ad-hoc solutions. The async architecture processes up to 1000 tickers per second on a single server — 3x faster than typical synchronous Python implementations. Maintenance savings from a unified format reach 50%.

How we ensure normalization accuracy

Unit tests with real raw-data samples from each exchange are mandatory. Exchanges sometimes change their API format without notice. A fixed set of fixtures with expected normalized outputs helps detect regressions quickly:

def test_binance_normalizer(): raw = {"s": "BTCUSDT", "c": "43250.50", "v": "28450.12", "T": 1704067200000} result = BinanceNormalizer().normalize_ticker(raw) assert result.symbol == "BTC/USDT" assert result.price == Decimal("43250.50") assert result.exchange == "binance" 

Additionally, we run integration tests against live exchange sandbox APIs daily in CI to catch API changes early.

Normalization steps checklist

• Audit exchange APIs: documentation, rate limits, formats.
• Design normalised data schema.
• Implement adapters for each exchange.
• Write unit and integration tests.
• Create integration documentation.
• Support for one month after delivery: refinements, consultations.

Step-by-step guide for adding a new adapter

1. Create a class inheriting from ExchangeNormalizer.
2. Implement normalize_symbol, normalize_ticker, normalize_order.
3. Write unit tests with raw-data samples.
4. Register the adapter in the normalizer factory.
5. Test integration on sandbox exchanges.
6. Deploy to production with error monitoring.

Timing and cost

Timelines range from 2 to 4 weeks per exchange; for a complex project with 5+ exchanges, 4 to 8 weeks. Cost is calculated individually after analyzing your requirements. For an accurate estimate, fill out a brief — we'll send a proposal with stages and timelines.

Order development of a normalization system tailored to your needs. Get a consultation from our engineer right now — we'll respond within one business day.