Building Blockchain Data Stream Processing with Kafka/Flink
We often encounter a situation where an Ethereum node in real-time generates about 2–5 MB of data per second during high network activity. That includes Transfer events, contract calls, and state changes. If your analytics system or trading engine fetches this data via periodic polling of an RPC node, you are working with stale data and missing events. For tasks where a 1–2 block delay is critical (arbitrage, liquidation monitoring, fraud detection), a streaming architecture with delivery guarantees is necessary. Our engineers build such systems turnkey — from topics to dashboards.
Why Streaming Blockchain Data Is Critical for DeFi
Arbitrage bots, liquidation monitoring, and MEV detection require latency under 500 ms from block arrival to decision-making. Polling an RPC node via JSON-RPC gives delays of seconds and no guarantee of event delivery. A streaming architecture on Kafka ensures data persistence with replay capability, and Flink enables sliding aggregations and complex pattern detection in real time.
Data Sources: From Node to Kafka
WebSocket Subscriptions vs Polling
The standard eth_subscribe("newHeads") via WebSocket notifies about new blocks without polling delay. However, WebSocket connections are unstable over long periods — reconnect with catch-up logic is needed:
func (s *NodeSubscriber) subscribeWithRecovery(ctx context.Context) error {
for {
lastBlock, _ := s.db.GetLastProcessedBlock()
// Catch up missed blocks on reconnect
if err := s.catchUpFromBlock(ctx, lastBlock+1); err != nil {
return err
}
// Subscribe to new blocks
sub, err := s.client.SubscribeNewHead(ctx, s.headers)
if err != nil {
time.Sleep(backoffDuration)
continue
}
select {
case err := <-sub.Err():
log.Warnf("subscription error: %v, reconnecting", err)
case <-ctx.Done():
return nil
}
}
}
Firehose Protocol (StreamingFast/Pinax)
For Ethereum and other EVM networks, the most efficient way to get raw data is Firehose (StreamingFast), which instruments the node at the binary level and exports blocks in protobuf with minimal latency. Throughput is an order of magnitude higher than JSON-RPC. For projects requiring full historical replay, Firehose plus flat files in S3/GCS allows reproducing any block range without re-syncing the node.
Kafka as Transport Layer
Kafka is a log-based queue. Unlike RabbitMQ/Redis Streams, Kafka persists all messages for a configured retention (days, weeks), allowing consumers to re-read data. This is critical for blockchain analytics: a new consumer group can read the entire event history without touching the node.
Topic topology for a blockchain pipeline:
raw.blocks → raw blocks (partitioned by block_number % N)
raw.transactions → all transactions
raw.logs → all event logs
decoded.transfers → decoded ERC-20 Transfer events
decoded.swaps → decoded Swap events (Uniswap, Curve, etc.)
alerts.large-txns → transactions > threshold
analytics.prices → aggregated price data
Partitioning strategy matters: for specific contract events, partition by contractAddress (guarantees ordering). For transactions, partition by from address or blockNumber.
Apache Flink: Stateful Stream Processing
Flink is the right tool for tasks requiring state: sliding aggregations, stream joins, temporal pattern detection. Spark Streaming is batching disguised as streaming (micro-batches). Flink is true event-time processing.
On-the-Fly ABI Decoding
Incoming logs are raw hex data. A Flink job must decode them into typed events:
public class LogDecoderFunction extends RichFlatMapFunction<RawLog, DecodedEvent> {
private Map<String, ContractABI> abiRegistry;
@Override
public void flatMap(RawLog log, Collector<DecodedEvent> out) {
String contractAddress = log.getAddress().toLowerCase();
ContractABI abi = abiRegistry.get(contractAddress);
if (abi == null) return; // unknown contract
String topic0 = log.getTopics().get(0);
EventDefinition eventDef = abi.findEventBySignatureHash(topic0);
if (eventDef != null) {
DecodedEvent decoded = AbiDecoder.decode(eventDef, log);
out.collect(decoded);
}
}
}
The ABI registry is loaded from PostgreSQL/Redis at job start and updated via Broadcast State pattern — no job restart when new contracts are added.
Temporal Windows and Aggregations
Task: compute 5-minute VWAP (Volume Weighted Average Price) from Uniswap V3 swaps in real time.
DataStream<SwapEvent> swaps = source
.filter(e -> e.getType().equals("Swap"))
.map(e -> (SwapEvent) e);
DataStream<VWAPResult> vwap = swaps
.keyBy(SwapEvent::getPoolAddress)
.window(TumblingEventTimeWindows.of(Time.minutes(5)))
.aggregate(new VWAPAggregator(), new VWAPWindowFunction());
Event time vs processing time — a fundamental choice. Event time (block time) gives deterministic results when replaying history. Processing time is faster but yields different results on replay.
Watermarks for handling late events — blockchain transactions may arrive in Kafka with slight delay:
WatermarkStrategy.<RawLog>forBoundedOutOfOrderness(Duration.ofSeconds(10))
.withTimestampAssigner((log, ts) -> log.getBlockTimestamp() * 1000L)
Complex Patterns: CEP for Anomaly Detection
Flink CEP (Complex Event Processing) allows describing event sequences. Task: detect a sandwich attack — front-run transaction, victim, back-run transaction within one block.
Pattern<DecodedEvent, ?> sandwichPattern = Pattern
.<DecodedEvent>begin("frontrun")
.where(e -> e.isSwap() && e.getGasPrice() > threshold)
.next("victim")
.where(e -> e.isSwap() && samePool(e, "frontrun"))
.next("backrun")
.where(e -> e.isSwap() && samePool(e, "frontrun")
&& e.getSender().equals(frontrunSender(e)))
.within(Time.seconds(12)); // within one block
State Backend and Fault Tolerance
How We Ensure Exactly-Once Delivery?
Flink checkpoint — snapshot of all operator state to S3/HDFS. On failure, recovery from the last checkpoint, Kafka consumer offset saved atomically with state. This guarantees exactly-once semantics for most operators.
RocksDB state backend — mandatory for production with large state (millions of keys). In-memory backend doesn’t scale.
Details on checkpointing
Checkpointing interval of 60 seconds balances performance and recovery. On failure, recovery takes no more than 2 minutes.Monitoring and Dead Letter Queues
Unprocessed events (unknown ABI, parsing error, unexpected format) cannot simply be dropped. Dead letter queue (DLQ) into a separate Kafka topic preserving the original message and stack trace — standard pattern.
Metrics: Flink + Prometheus + Grafana: lag per topic, operator throughput, backpressure in job graph. Backpressure is the first indicator that downstream can’t keep up.
Typical Use Cases and Latency
| Use Case | Acceptable Latency | Tool |
|---|---|---|
| MEV bot / arbitrage | < 100 ms | WebSocket → in-process |
| Liquidation monitoring | < 1 sec | Kafka + Flink CEP |
| Real-time DeFi analytics | 1–5 sec | Kafka + Flink aggregations |
| On-chain analytics / BI | < 1 min | Kafka + Flink → ClickHouse |
| Historical analysis | no limit | Firehose → S3 → Spark/dbt |
Comparison of Stream Processing Tools
| Tool | Approach | Delivery Guarantee | Latency |
|---|---|---|---|
| Apache Flink | True streaming, event-time | Exactly-once | < 100 ms |
| Kafka Streams | Stream-table duality | At-least-once | < 100 ms |
| Spark Streaming | Micro-batches | Exactly-once (via checkpoint) | ~ 1 sec |
| Akka Streams | Reactive streams | At-most-once | < 50 ms |
Infrastructure and Stack
Minimum production cluster: 3 Kafka brokers (3 replicas for durability), Flink cluster with 1 JobManager + 3–5 TaskManager pods in Kubernetes. Result storage: ClickHouse for analytical queries (columnar, fast aggregations on large volumes) or PostgreSQL + TimescaleDB for time-series metrics.
Managed services reduce operational load: Confluent Cloud (Kafka), Amazon Kinesis (alternative for AWS-native stack). For on-premise or compliance requirements — own cluster.
What’s Included in System Development
- Architecture of streaming pipeline from sources to storage
- Kafka setup: topics, partitioning, retention policies
- Flink job development: ABI decoding, aggregations, CEP patterns
- Monitoring and alerting: Prometheus + Grafana dashboards
- Documentation and team training
- Post-launch support (per SLA)
Our team has 7+ years of experience building high-load systems for Crypto and DeFi, having delivered 30+ projects. We’re ready to assess your project — get in touch. Evaluation takes 2 business days.