Multi-Chain Data Aggregation: Architecture & Implementation

Multi-Chain Data Aggregation: Architecture & Implementation

The task looks simple: "collect data from multiple blockchains." In practice, it's one of the most technically complex tasks in Web3 infrastructure. Each network has its own data model, finality logic, RPC API, rate limits, and specific quirks. Ethereum lives in UTC with ~12-second blocks, Solana produces ~400ms slots and treats confirmations differently, TON has a sharded architecture where "block" is a loose concept. Collecting all this into a single API with consistent data is no trivial engineering feat.

Our team brings over 5 years of experience in blockchain infrastructure development and has built aggregators for 15+ networks. We know all the pitfalls: from non-obvious reorganizations to economizing on RPC calls. We guarantee 99.9% uptime and offer a free assessment for your project.

Challenges in Unifying Data from Different Networks

Different Data Models

EVM networks (Ethereum, Arbitrum, Polygon, BSC) share a common model: blocks, transactions, receipts with logs. But even here, there are differences:

• Arbitrum adds l1BlockNumber and specific system transactions (sequencer batch submissions)
• Optimism/Base have depositedTx type for L1→L2 transactions that lack a standard from
• zkSync Era uses native AA — no distinction between EOA and contracts; all accounts are contracts

Solana is a completely different paradigm: there is no "transaction called a contract method" — instead, "instructions in a transaction are passed to programs." Decoding requires an ABI equivalent: IDL (Interface Definition Language, Anchor format).

UTXO models (Bitcoin, Litecoin) are fundamentally different: no account balances, only unspent outputs. An "address balance" is the sum of all UTXOs where that address is an output.

Different Finality Semantics

If the system doesn't account for this, data will be incorrect: a transaction may appear "final" based on confirmation count but then get reorganized.

According to the Ethereum specification after Merge, reorganization depth rarely exceeds 2 blocks.

Aggregator Architecture

Collector Layer (Chain Collectors)

Each collector is an isolated service responsible for one network with a common interface:

interface ChainCollector { getLatestBlock(): Promise<UnifiedBlock>; getBlockRange(from: bigint, to: bigint): Promise<UnifiedBlock[]>; getTransactionsByAddress(address: string, fromBlock: bigint): Promise<UnifiedTx[]>; subscribeNewBlocks(callback: (block: UnifiedBlock) => void): Unsubscribe; } 

Unified types normalize each network's specifics:

interface UnifiedTx { chain: ChainId; hash: string; blockNumber: bigint; timestamp: number; // unix from: string; // normalized lowercase hex for EVM, base58 for Solana to: string | null; value: bigint; // in smallest native token units status: 'success' | 'failed' | 'pending'; finality: 'unconfirmed' | 'safe' | 'finalized'; raw: unknown; // original network data } 

Node & Provider Management

Problem: public RPCs are unreliable, rate limits are unpredictable, Alchemy/Infura get expensive at scale.

Strategy: tiered provider pool

Primary: Own nodes (Geth+Lighthouse, Reth for archive) ↓ failover Secondary: Alchemy / QuickNode (premium tier) ↓ failover Tertiary: Infura / public RPCs (non-critical only) 

Circuit breaker on each provider: if error rate > 5% over 60 sec or latency > 2x p99 baseline — remove provider from rotation, health check every 30 sec.

For archive data (historical blocks > 128 blocks back on Ethereum) an archive node is needed — that's a separate story. Running an Ethereum archive node on a cloud provider costs approximately $400-600/month, but using a tiered provider pool can reduce monthly RPC costs by up to 50%, saving thousands for high-volume applications. Erigon takes ~3TB for a full Ethereum archive, Reth slightly less. For most projects, it's cheaper to use Alchemy Archive or QuickNode Archive than to host your own node.

Building your own aggregator can save up to 50% on infrastructure costs compared to using third-party APIs, which can translate to thousands of dollars per month for high-throughput applications.

Why Own Node or Provider Pool Matters?

RPC reliability directly affects data consistency. Without redundancy, you risk data collection lag or block loss during reorgs. Own nodes reduce long-term costs: each RPC call costs money, and with millions of transactions, savings can reach up to 50% of the infrastructure budget. We recommend a tiered approach to balance cost and reliability.

Normalization & Transformation Layer

Raw blockchain data is rarely needed as-is. Common transformations: Decoding ERC-20 Transfer events

const ERC20_TRANSFER_TOPIC = "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"; function decodeTransfer(log: Log): TokenTransfer | null { if (log.topics[0] !== ERC20_TRANSFER_TOPIC) return null; return { token: log.address, from: `0x${log.topics[1].slice(26)}`, to: `0x${log.topics[2].slice(26)}`, amount: BigInt(log.data), }; } 

Enriching with token data: for each log.address, we need symbol, decimals, USD price. We cache token metadata in Redis with TTL 24h, and update prices every 30 sec from CoinGecko/CoinMarketCap.

Cross-chain aggregation: to show "total address balance across all networks in USD", we need to normalize different decimals, convert via price feeds, and handle wrapped versions of the same token (USDC on Ethereum ≠ USDC.e on Arbitrum).

Storage Layer

For hot data (last 7–30 days): PostgreSQL with partitioning by chain_id + date. Indexes on (chain_id, address, block_number) and (chain_id, tx_hash). TimescaleDB hyper tables if data volume is high — automatic compression of old partitions.

For cold data (archive): ClickHouse — a columnar database, an order of magnitude more efficient than PostgreSQL for analytical queries over large periods. ClickHouse is 10-100x faster than PostgreSQL for analytical queries over large datasets. A query "all USDC transactions > $10k over the past year across all EVM networks" on 100M+ rows — ClickHouse returns in seconds, PostgreSQL in minutes.

For address/hash search: ElasticSearch or simply PostgreSQL with LIKE — for exact matches, a hash index suffices.

Ensuring Data Consistency During Reorganizations

This is the hardest part. Algorithm:

1. Each block is saved with is_canonical = true and parent_hash
2. A new block with the same block_number but different hash indicates a potential reorg
3. Follow parent_hash back until a common ancestor is found
4. Mark all blocks on the "old" branch is_canonical = false, add blocks of the "new" branch
5. Output API data is always filtered by is_canonical = true
6. Webhooks/downstream systems receive tx.orphaned events for reverted transactions

For Ethereum, reorg depth is extremely rare > 2 blocks post-Merge. For Polygon PoS, we've seen reorgs of 30+ blocks. Observation buffer: 128 blocks for EVM networks.

API Layer

REST + WebSocket for real-time:

GET /v1/address/{address}/transactions?chains=eth,arb,polygon&limit=50 GET /v1/tx/{chain}/{hash} GET /v1/address/{address}/token-balances?chains=eth,bsc WS /v1/subscribe?address={addr}&chains=eth,arb&events=transfer,swap 

GraphQL is convenient if clients need query flexibility: one request gets transactions + balances + token metadata. But it adds backend complexity — N+1 problems, need DataLoader.

Rate limiting: per-API-key, sliding window, separate limits for REST and WebSocket (WebSocket connections are more expensive). Redis + Lua script for atomic increments.

Monitoring & Operations

Critical metrics:

• Collector lag — difference between latest block timestamp on the network and the time that block was processed. Alert if lag > 2 minutes.
• Reorg depth — maximum reorg depth in the last 24h. Alert if depth > 10.
• RPC error rate — per provider and method. Alert if > 1%.
• Queue depth — if the processor can't keep up with the collector, queue grows. Alert if depth > 10k messages.

Grafana dashboard with per-chain panels: current block, lag, TPS, error rate.

Tech Stack

What's Included

1. System architecture diagram
2. Source code for collectors and API
3. Deployment and operations documentation
4. Monitoring and alerts (Grafana dashboards)
5. 2 months of technical support
6. Training for the client's team

Timeline for MVP (3–4 EVM networks, no archive, REST API): 8–12 weeks. Full system with 10+ networks, ClickHouse, WebSocket, monitoring: 5–7 months.

Own nodes are 50% cheaper than using third-party RPC providers for high-throughput networks. Savings on RPC providers through query optimization and own nodes can reach up to 50%.

Contact us to discuss your project. We'll assess scope and cost for free. Get a consultation within a day. Order a turnkey aggregation system development.