Why streaming data processing breaks under peak loads
Streaming data processing breaks not at the very peak, but slightly earlier — when the queue is still holding, yet event processing latency has already exceeded the acceptable threshold.
Peak load on the pipeline arrives unevenly: external sources and internal services emit events evenly on average, but in short intervals they can be several times faster. The pipeline holds as long as the read rate matches the write rate.
Once the consumer falls behind, the lag begins to grow by itself: draining the accumulated queue requires more resources than processing a fresh stream, and the gap no longer resolves.
Next, event loss in the queue appears, and almost never as a broker crash.
An event disappears more quietly: the topic retention period expired while the backlog was being drained; the offset was committed before the result was actually written; the consumer crashed between reading and saving.
Externally the system looks alive, metrics are normal, yet some events never arrived — and that is discovered during data reconciliation, not by a monitoring alert.
The flip side of delivery guarantees is event duplication. When consumers are rebalanced within a group or a message is redelivered, a single event enters processing twice, and the windowed aggregate counts it as two.
Such a duplicate in a report is indistinguishable from a real spike, so a decision is made based on a number that never existed.
The third class of failures is prediction divergence during recomputation.
If the model was updated but events from the accumulated queue were processed further with the old state, the same event receives different scores: arrival order, late records, and the time by which features were collected all have an effect. Reports stop reconciling with each other, and trust in the system falls faster than the lag grows.
All three failures are about the pipeline, not the model: they manifest only under load and are almost never reproduced on a test stream. Below we discuss how this is addressed at the level of the pipeline and control over it.
What a streaming pipeline delivers: measurable effect
A streaming pipeline computes features and calls the model on every event, not on a schedule. Below are the effects visible in operation: response time, eliminated failures, removed manual work, and transparency.
Reduced prediction latency
Features are collected in a sliding window, the model responds on the same event, and the result immediately goes to the data mart and to the consuming service.
Batch recomputation gives an answer minutes or hours later, while streaming — within a second of the event, and latency does not grow with stream volume.
Eliminated class of failures: losses and duplicates
The transport is Apache Kafka: delivery acknowledgment and idempotent writing by event key eliminate a whole class of failures — messages are not lost when a consumer restarts and do not duplicate a prediction.
After a node crash, the pipeline continues from the saved position, without manual reconciliation or reprocessing of accumulated data.
How much manual work automatic recomputation removes
Feature marts and aggregates are recomputed automatically — per event, not by an engineer's trigger. Previously, a discrepancy was looked for by eye and the job was restarted manually: hours of work and a window during which decisions were made based on stale numbers. Now the mart reconciles itself, and only cases requiring intervention enter the log.
Pipeline observability — consumer lag, latency of each segment, share of rejected records — turns failure analysis into reading metrics. The error is visible before it reaches the user.
Event delivery modes: what to choose for the task
The processing mode is chosen not by fashion, but by acceptable latency and the cost of a missed event. If a data-driven decision is needed once a day, a streaming pipeline adds failure points and operational cost with no benefit to the result.
Scheduled batch mode
Recomputation reads the accumulated slice, computes aggregates, and writes the result as a whole. The limitation: latency equals the launch interval, and a data spike shifts the next window. For reporting, feature recomputation, and model training, this is the most predictable option.
Micro-batches and pure streaming: where the boundary is
A micro-batch collects events over a short window and emits them as a batch: latency is seconds, and the cost is close to batch.
A streaming design processes each event as it arrives, but requires stored state, handling of late events, and redelivery — without this, the result diverges on failure.
| Option | When it fits | Limitations |
|---|---|---|
| Scheduled batch processing | Reporting, feature recomputation, training | Latency equals the launch interval |
| Micro-batch mode | Metrics and marts with second-level latency | Late events complicate the logic |
| Pure streaming design | Scoring, anomaly response | State, retries, operational cost |
| Hybrid pipeline | Fast response plus background recomputation | Reconciliation of streaming and batch results is required |
A hybrid pipeline removes the limitations of a streaming design: the stream provides a fast response, while scheduled background recomputation reconciles and corrects it.
We build the choice of processing mode into the architecture in advance — migrating a service from batch to streaming later is more expensive than designing for a hybrid from the start.
How we deploy a streaming pipeline: stages of work
Until the load profile is clear, any streaming pipeline design remains a guess. That is why the stages go from data to code: each next step relies on an artifact from the previous one, not on assumptions about how the stream behaves.
Audit of sources and load profile
- Data source audit — inventory of topics, message formats, partitioning keys, and the share of “dirty” records. The output is a source map with notes about delays and schema changes that break backward compatibility.
- Load profile — we capture the real stream: peaks, seasonality, distribution by keys. From this we calculate the number of parallel processors and memory headroom.
- Streaming pipeline design — we fix the boundaries: where we read, where we enrich, where we call the model, where we store. We separately describe delivery semantics and the procedure for replaying the window on failure.
Testing on shadow traffic before cutover
- Debugging on historical data — run the pipeline on the accumulated stream and reconcile with the current solution; we investigate discrepancies before cutover.
- Load testing — we feed a stream above the estimated peak and see exactly where degradation occurs: queue growth, processing lag, share of on-time responses.
- Shadow traffic — the new pipeline computes live data in parallel with the production branch, and the result goes nowhere. This way we test behavior on rare events and make sure discrepancies stay within tolerance.
- Handover to operation — traffic switchover, handover to the team of configurations, alert rules, and instructions for rolling back to the previous pipeline.
What you keep: diagrams, procedures, configurations
We hand over the pipeline not merely “working,” but maintainable: the client’s engineer should be able to start the stream, add a field to an event, and recover from a failure without waiting for us.
That is why the handover includes not only code, but also a set of artifacts that allow streaming data processing to be maintained and evolved within the team.
Stream diagrams and event format descriptions
- Stream map: event sources, queues and partitions, consumers, the event path from ingestion to storage write and callbacks.
- Versioned event format description: required and optional fields, units of measurement, compatibility rules when adding a field.
- Processing configurations: aggregation windows, allowed lag, retries on errors, state checkpoints, storage depth before flush.
Failure response procedures
- Incident levels and response time for each: stream lag, event loss, model failure at inference.
- Procedure: how to stop ingestion, replay a segment from the last checkpoint, roll back a schema or model version.
- Data access rules: roles, what access to each stream is composed of, where keys are stored and how they are rotated.
- Pipeline documentation: deployment diagram, dependencies, stream owners, change procedure.
Each item is accompanied by a short checklist — the client accepts the work themselves, not on our word.
Case study: how we reduced latency under peak load
The client is a platform where the model must respond to every event from devices within the same request.
At peak, the stream grew threefold, and the pipeline could not keep up: predictions went into a queue, some events were lost when a node restarted, and after a failure recovery stretched to tens of minutes — for the business, this looked like “the service is blinking.”
What it was before and what it became after
We started the analysis with the stream: it was read by a single consumer, so there was nothing to scale.
We split processing across independent Kafka consumers with a partitioning key by device ID — event order for a single object is preserved, while load is distributed across nodes.
Window state was moved to fault-tolerant storage, and a node restart no longer meant losing accumulated context.
| Metric | Before | After |
|---|---|---|
| Prediction latency at peak | 12x increase | 1.4x increase |
| Share of lost events | 3–4% during peak hour | 0.02% |
| Pipeline recovery after node failure | 20–35 minutes | up to 40 seconds |
We made reprocessing idempotent: every event has a key, so replaying the log does not produce duplicates in storage or divergence of aggregates.
It was precisely this class of failures that caused double records after a failure and forced engineers to manually reconcile the marts.
How does streaming processing differ from batch processing?
Batch processing computes over an accumulated slice: data is collected, and the calculation is launched as a whole on a schedule. Streaming processing reads events as they arrive and updates the result continuously.
The difference is not in volume, but in result latency: in a batch pipeline it equals the launch interval; in a streaming one, it is the time it takes an event to pass through the window and the model.
Latency defines the boundary of applicability. As long as the answer arrives once a day, scoring at the moment of application, anomaly alerts, or an in-interface hint are impossible.
Streaming shifts response to seconds but pays for it with constant pipeline operation: it holds aggregate state between events and does not “sleep” between runs.
Recomputing historical data is a strength of the batch approach: we change the logic, rerun the period, and get a consistent picture.
In streaming, this is a separate operation — restore state at the required moment, bound the window, and make the write idempotent; otherwise reprocessing will double the counters and distort the mart.
Debugging complexity in streaming is higher for an objective reason. A batch calculation is reproducible: there is an input slice, a code version, and a result.
A streaming one depends on event order, late records, accumulated state, and the moment of failure — it needs an input event log, state checkpoints, and replay from the same position.
Data quality requirements diverge in the same way: in batch, a corrupt record is isolated, corrected, and recomputed; in streaming, an incompatible field breaks processing on the fly.
That is why pipelines are not set against each other but combined: streaming is responsible for freshness, batch for completeness and reconciliation. The choice is dictated not by data volume, but by acceptable result latency and the cost of an error.
Common technical concerns before starting
Before starting, engineering teams ask the same questions: what happens to late events, how to avoid duplicating results on redelivery, and whether the pipeline will withstand stream growth. Below are short answers on how these risks are addressed in streaming processing in practice.
What happens to late events?
We calculate the aggregation window by the event time itself, not by its arrival time, with an explicitly defined allowed lag.
Anything that arrives after the boundary is not discarded: such records go into a separate backfill stream, and the final marts are updated — the data tail is not lost.
How is idempotent processing ensured?
Each message carries a key, and the result write is performed with a check of that key, so redelivery does not create a duplicate. On restart, the pipeline continues from the last confirmed position — a failure does not turn into doubled values in reporting.
How is data quality controlled?
At the input, we set schema, range, and required field checks, and anomalies do not bring down the stream but are marked and sent to quarantine. The share of rejected records and the reasons go into monitoring, so source degradation is visible before it reaches the models.
Will the pipeline withstand load growth?
Consumers scale by the number of partitions, so throughput grows by adding processors without rewriting the logic. We run the peak load profile in advance and record at what rate expansion is needed.
How do you launch without risk to production?
We proceed in stages: first a shadow run on a copy of the stream and reconciliation of results with the current pipeline, then a gradual transfer of a share of traffic with the ability to switch back. Rollback means returning to the previous route, not restoring from backup.
Let's discuss the task and come back with an estimate and plan
A substantive conversation does not require a finished document — a description of how the data stream lives now is enough. The more specific the inputs, the more accurate the estimate and work plan.
- event sources: applications, sensors, queues, databases — where data comes from, in what format, and at what frequency;
- description of the current design: where the model is currently computed, where stream state is stored, which segments no longer handle the load;
- load constraints: peak number of events per second, acceptable response latency, storage volumes;
- decision quality requirements: what exactly the model predicts and what error you consider unacceptable;
- infrastructure and access: your own pipeline or cloud, data storage and transfer rules;
- project start criteria: what result at the first stage you would consider successful.
In response, we send an estimate and work plan: a breakdown by stages with durations, a data flow diagram, a list of risks and rollback points, the team composition, and the area of responsibility on your side.
If the inputs show that part of the pipeline does not need to be redone, we will say so directly — this reduces the scope of work.
You can discuss the task even without a full description: send what you have, and we will clarify the rest with questions.
