On-Chain Data Pipeline Engineering for ML

We often encounter the request: "We want to predict whale activity" or "We need an on-chain credit risk model." Behind that is an engineering problem most teams underestimate: raw blockchain data is not directly usable for ML models. Block structures, raw hex-encoded calldata, bytes20 addresses — these are not features; they're raw materials. Between an RPC node and a training dataset lie several weeks of infrastructure work. For example, to build a liquidity outflow model for a DeFi protocol, you need not only Transfer events but also internal calls, trace information, and normalized timestamps to a single time zone. Each of these operations requires a separate pipeline with error control and reproducibility. In practice, without the right pipeline, you risk generating garbage features that degrade model quality.

Our on-chain ML infrastructure handles everything from Ethereum data pipeline to wallet profiling and MEV detection, using point-in-time features and Feast feature store for blockchain data processing. We specialize in building on-chain data pipelines for ML, reducing infrastructure costs and ensuring point-in-time correctness.

Why raw blockchain data is unsuitable for ML

A raw Transfer log is three bytes32 values plus data bytes. To turn it into ML features, you need:

• Decoding — ABI decoding of topics and data
• Address normalization — uint256 → checksummed hex, label mapping (exchanges, protocols, MEV bots)
• Monetary normalization — value / 10^decimals, conversion to USD via historical price feed
• Entity resolution — one EOA may have hundreds of transactions but be a single economic agent; smart contracts — proxies, implementations, multisigs

Skipping any step leads to garbage features.

According to the Ethereum Foundation documentation, integration with an archive node via the trace API provides full history of internal transactions.

Data sources: from RPC to specialized providers

Public RPC (eth_getLogs, eth_getBlockByNumber) is the most accessible but least suitable for ML. Its limitations: rate limits (Infura/Alchemy — 10-333 req/s on paid plans), no internal transactions without the trace_ namespace, no pre/post state without an archive node. An archive node with trace API gives full history but requires Erigon with ~2.5 TB disk space for Ethereum mainnet and 3-5 days sync. The trace_ formats differ between Erigon and Geth/Besu — the parser must be adapted. Firehose (StreamingFast/The Graph) exports each block with the call tree and state diffs in <500ms, achieving 100k+ blocks per minute — 20-100x faster than RPC. Specialized providers (Nansen, Dune, Flipside, Allium) offer pre-normalized tables but with 1-24 hour update latency and limited schema control. For production ML, we recommend combining: Firehose for historical loading and an archive node for real-time streaming.

How point-in-time correctness is guaranteed at the backend

This is the key problem. Features must be computed only from data available before the prediction moment. A typical mistake: using total_tx_count of an address instead of tx_count_at_time_T. Pattern: temporal consistency features. Each row in the feature store has entity_id, feature_timestamp, and feature_value. When generating training data, join by entity_id and feature_timestamp <= label_timestamp.

-- Temporal join SELECT l.wallet_address, l.label, l.label_timestamp, f.tx_count, f.unique_contracts, f.volume_usd_30d FROM labels l ASOF JOIN wallet_features f ON l.wallet_address = f.wallet_address AND f.feature_timestamp <= l.label_timestamp 

ASOF JOIN is native in ClickHouse and TimescaleDB; in PostgreSQL it is emulated via LATERAL.

Offline store — historical features for training. ClickHouse or Parquet on S3 with Hive-partitioning by date. Online store — current features for inference. Redis Hash structures: HGETALL wallet:{address}:features. Updated with each new block for active addresses.

Production pipeline architecture

Ingestion layer

We recommend an event-driven architecture with hot and cold path separation:

[Archive Node / Firehose] ↓ [Kafka / Redpanda] ← hot path: < 1s latency ↓ [Stream Processor] ← Flink or custom consumer / \ [Raw Store] [Feature Store] ← cold: S3/Parquet, hot: Redis/Feast 

Kafka topic per chain, key = block_number:log_index. This guarantees order and allows replay on processing errors. Retention depends on the task: 7 days for real-time features, full archive in S3 for retraining. For Ethereum mainnet: ~6000 transactions/block × ~6500 blocks/day = ~39M transactions/day. Average transaction size with trace ~2KB = ~75GB/day raw data. Plan storage accordingly.

Feature engineering and stores

This is the most labor-intensive part. Typical blockchain features for various ML tasks: Wallet profiling (DeFi credit scoring, Sybil detection):

MEV detection:

• Sandwich attack pattern: three TXs in one block, same address, surrounding target TX
• Arbitrage: cyclic token transfers returning to sender within one TX
• Flashloan: FlashLoan event + position delta = 0 at block end

Whale activity prediction:

• Large transfers from exchange deposit addresses → sell pressure probability
• Accumulation pattern: multiple small purchases from different addresses → one recipient

# Example feature engineering for wallet scoring import polars as pl def compute_wallet_features(txs: pl.DataFrame) -> pl.DataFrame: return txs.group_by("from_address").agg([ pl.col("block_number").min().alias("first_seen_block"), pl.col("block_number").max().alias("last_seen_block"), pl.count("hash").alias("tx_count"), pl.col("to_address").n_unique().alias("unique_contracts"), pl.col("gas_price").quantile(0.5).alias("gas_price_median"), pl.col("value_usd").sum().alias("total_volume_usd"), pl.col("block_timestamp").diff().dt.total_seconds() .mean().alias("avg_interval_seconds"), ]) 

Polars over Pandas — speed difference on large datasets (millions of rows) is 5-20x.

Reorganization handling and MLOps

Reorgs at the ML data level are a serious issue. If features are computed from a block that later becomes orphaned, the training set contains unrealistic data. Solutions:

• Confirmation lag — index only blocks older than N blocks (usually 12-32 for finality on PoS Ethereum). Adds latency but solves the problem.
• Versioned features — store (entity, block_hash, features), mark orphaned entries on reorg. More complex but allows low-latency operation.

MLOps integration. The pipeline must integrate with your existing ML stack: Feature generation → training: export to Parquet/CSV for DVC or MLflow artifacts. Dataset versioning is critical — a model trained on data from a specific period must be reproducible. Inference pipeline: new block → compute delta features → update online store → trigger inference. Latency budget is typically 1-10 seconds from block to prediction. Model drift monitoring: blockchain data changes structurally (merges, new protocols, pattern shifts). We set up monitoring of input feature distribution — Evidently AI or custom.

Typical project phases

1. Data audit (1-2 weeks) — Identify required signals, their sources, and historical data availability. Prototype ingestor on a small block range.

2. Historical backfill (2-4 weeks) — Load historical data, normalize, label mapping. The most time-consuming phase.

3. Feature pipeline (2-3 weeks) — Implement feature engineering, temporal consistency logic, storage.

4. Real-time path (1-2 weeks) — Stream from node, online store, inference integration.

5. MLOps (1-2 weeks) — Drift monitoring, dataset versioning, automated retraining.

Total: 7-13 weeks to production-ready pipeline. Estimate varies heavily with number of chains, historical depth, and inference latency requirements.

What's included

• Architecture design for your specific problem
• Infrastructure setup (Kafka, ClickHouse, Redis)
• Feature engineering for target signals
• MLOps integration (MLflow, DVC)
• Documentation and team training

Founded in 2019, we have 5+ years of market experience and a track record of 20+ successful on-chain data projects. Our team has 10+ years of blockchain development experience. Using our on-chain data pipeline can reduce infrastructure costs by up to 40% — clients typically save $10,000–$20,000 per month. For example, one client saved $15,000/month after migrating to our system. Request an audit of your blockchain data — get a prototype pipeline in 2 weeks. Contact us for a consultation.