Building Blockchain Data Stream Processing with Kafka/Flink

Building Blockchain Data Stream Processing with Kafka/Flink

We often encounter a situation where an Ethereum node in real-time generates about 2–5 MB of data per second during high network activity. That includes Transfer events, contract calls, and state changes. If your analytics system or trading engine fetches this data via periodic polling of an RPC node, you are working with stale data and missing events. For tasks where a 1–2 block delay is critical (arbitrage, liquidation monitoring, fraud detection), a streaming architecture with delivery guarantees is necessary. Our engineers build such systems turnkey — from topics to dashboards.

Why Streaming Blockchain Data Is Critical for DeFi

Arbitrage bots, liquidation monitoring, and MEV detection require latency under 500 ms from block arrival to decision-making. Polling an RPC node via JSON-RPC gives delays of seconds and no guarantee of event delivery. A streaming architecture on Kafka ensures data persistence with replay capability, and Flink enables sliding aggregations and complex pattern detection in real time.

Data Sources: From Node to Kafka

WebSocket Subscriptions vs Polling

The standard eth_subscribe("newHeads") via WebSocket notifies about new blocks without polling delay. However, WebSocket connections are unstable over long periods — reconnect with catch-up logic is needed:

func (s *NodeSubscriber) subscribeWithRecovery(ctx context.Context) error { for { lastBlock, _ := s.db.GetLastProcessedBlock() // Catch up missed blocks on reconnect if err := s.catchUpFromBlock(ctx, lastBlock+1); err != nil { return err } // Subscribe to new blocks sub, err := s.client.SubscribeNewHead(ctx, s.headers) if err != nil { time.Sleep(backoffDuration) continue } select { case err := <-sub.Err(): log.Warnf("subscription error: %v, reconnecting", err) case <-ctx.Done(): return nil } } } 

Firehose Protocol (StreamingFast/Pinax)

For Ethereum and other EVM networks, the most efficient way to get raw data is Firehose (StreamingFast), which instruments the node at the binary level and exports blocks in protobuf with minimal latency. Throughput is an order of magnitude higher than JSON-RPC. For projects requiring full historical replay, Firehose plus flat files in S3/GCS allows reproducing any block range without re-syncing the node.

Kafka as Transport Layer

Kafka is a log-based queue. Unlike RabbitMQ/Redis Streams, Kafka persists all messages for a configured retention (days, weeks), allowing consumers to re-read data. This is critical for blockchain analytics: a new consumer group can read the entire event history without touching the node.

Topic topology for a blockchain pipeline:

raw.blocks → raw blocks (partitioned by block_number % N) raw.transactions → all transactions raw.logs → all event logs decoded.transfers → decoded ERC-20 Transfer events decoded.swaps → decoded Swap events (Uniswap, Curve, etc.) alerts.large-txns → transactions > threshold analytics.prices → aggregated price data 

Partitioning strategy matters: for specific contract events, partition by contractAddress (guarantees ordering). For transactions, partition by from address or blockNumber.

Apache Flink: Stateful Stream Processing

Flink is the right tool for tasks requiring state: sliding aggregations, stream joins, temporal pattern detection. Spark Streaming is batching disguised as streaming (micro-batches). Flink is true event-time processing.

On-the-Fly ABI Decoding

Incoming logs are raw hex data. A Flink job must decode them into typed events:

public class LogDecoderFunction extends RichFlatMapFunction<RawLog, DecodedEvent> { private Map<String, ContractABI> abiRegistry; @Override public void flatMap(RawLog log, Collector<DecodedEvent> out) { String contractAddress = log.getAddress().toLowerCase(); ContractABI abi = abiRegistry.get(contractAddress); if (abi == null) return; // unknown contract String topic0 = log.getTopics().get(0); EventDefinition eventDef = abi.findEventBySignatureHash(topic0); if (eventDef != null) { DecodedEvent decoded = AbiDecoder.decode(eventDef, log); out.collect(decoded); } } } 

The ABI registry is loaded from PostgreSQL/Redis at job start and updated via Broadcast State pattern — no job restart when new contracts are added.

Temporal Windows and Aggregations

Task: compute 5-minute VWAP (Volume Weighted Average Price) from Uniswap V3 swaps in real time.

DataStream<SwapEvent> swaps = source .filter(e -> e.getType().equals("Swap")) .map(e -> (SwapEvent) e); DataStream<VWAPResult> vwap = swaps .keyBy(SwapEvent::getPoolAddress) .window(TumblingEventTimeWindows.of(Time.minutes(5))) .aggregate(new VWAPAggregator(), new VWAPWindowFunction()); 

Event time vs processing time — a fundamental choice. Event time (block time) gives deterministic results when replaying history. Processing time is faster but yields different results on replay.

Watermarks for handling late events — blockchain transactions may arrive in Kafka with slight delay:

WatermarkStrategy.<RawLog>forBoundedOutOfOrderness(Duration.ofSeconds(10)) .withTimestampAssigner((log, ts) -> log.getBlockTimestamp() * 1000L) 

Complex Patterns: CEP for Anomaly Detection

Flink CEP (Complex Event Processing) allows describing event sequences. Task: detect a sandwich attack — front-run transaction, victim, back-run transaction within one block.

Pattern<DecodedEvent, ?> sandwichPattern = Pattern .<DecodedEvent>begin("frontrun") .where(e -> e.isSwap() && e.getGasPrice() > threshold) .next("victim") .where(e -> e.isSwap() && samePool(e, "frontrun")) .next("backrun") .where(e -> e.isSwap() && samePool(e, "frontrun") && e.getSender().equals(frontrunSender(e))) .within(Time.seconds(12)); // within one block 

State Backend and Fault Tolerance

How We Ensure Exactly-Once Delivery?

Flink checkpoint — snapshot of all operator state to S3/HDFS. On failure, recovery from the last checkpoint, Kafka consumer offset saved atomically with state. This guarantees exactly-once semantics for most operators.

RocksDB state backend — mandatory for production with large state (millions of keys). In-memory backend doesn’t scale.

Monitoring and Dead Letter Queues

Unprocessed events (unknown ABI, parsing error, unexpected format) cannot simply be dropped. Dead letter queue (DLQ) into a separate Kafka topic preserving the original message and stack trace — standard pattern.

Metrics: Flink + Prometheus + Grafana: lag per topic, operator throughput, backpressure in job graph. Backpressure is the first indicator that downstream can’t keep up.

Typical Use Cases and Latency

Comparison of Stream Processing Tools

Infrastructure and Stack

Minimum production cluster: 3 Kafka brokers (3 replicas for durability), Flink cluster with 1 JobManager + 3–5 TaskManager pods in Kubernetes. Result storage: ClickHouse for analytical queries (columnar, fast aggregations on large volumes) or PostgreSQL + TimescaleDB for time-series metrics.

Managed services reduce operational load: Confluent Cloud (Kafka), Amazon Kinesis (alternative for AWS-native stack). For on-premise or compliance requirements — own cluster.

What’s Included in System Development

• Architecture of streaming pipeline from sources to storage
• Kafka setup: topics, partitioning, retention policies
• Flink job development: ABI decoding, aggregations, CEP patterns
• Monitoring and alerting: Prometheus + Grafana dashboards
• Documentation and team training
• Post-launch support (per SLA)

Our team has 7+ years of experience building high-load systems for Crypto and DeFi, having delivered 30+ projects. We’re ready to assess your project — get in touch. Evaluation takes 2 business days.