Apache Spark MLlib: where big data processing breaks
Distributed big data processing breaks not on model math, but on how data is laid out across nodes. The same calculation on a small sample takes minutes, while on the full volume it stretches to hours.
At the same time, the cause is almost never in Apache Spark MLlib itself, but in how computations are organized around it.
The most common failure is data skew. One key or a narrow range of values creates a block that is tens of times heavier than the others: some nodes finish their work and sit idle, while the remaining ones go into disk spilling and constant exchanges with each other.
The job stops being predictable in time, and regular recalculation can no longer be scheduled around it.
Second place is job failure at peak. When many times more data arrives than usual, computation starts from scratch, and recalculation time grows nonlinearly: an hour turns into a night.
Then manual “catch-ups,” reruns, and discrepancies between data marts begin, which someone later reconciles by hand.
The real bottleneck in processing is rarely in the calculations themselves. More often it is feature preparation and writing intermediate results: heavy transformations are duplicated on every run, and the result is not saved anywhere as a reusable layer.
We examine how this is addressed — from laying out data across nodes to restarting from the last successful point.
Implementation effect: speed, resilience, predictability
Moving to distributed computation changes measurable values, not general impressions.
Data marts are assembled not by one machine but by partitions: Apache Spark MLlib trains the model on nodes and aggregates the result, so sample size stops hitting the memory limit of one server.
A calculation that previously took the entire night window fits into the pause between scheduled jobs and leaves headroom for a rerun.
Fault tolerance is visible in the share of successful runs. Loss of a node no longer brings down the job: the scheduler retries only the lost partitions, and the lineage log shows which steps need recomputation.
A restart after failure becomes idempotent — state is restored from a checkpoint, not assembled manually.
Recovery time is measured in minutes instead of hours of log parsing. A whole class of incidents disappears: out-of-memory on final aggregation, hanging on a bottleneck node, full restart due to an error at the last step.
An unfinished calculation is either completed or rolled back to the previous data mart without manual edits.
The practical effect is that the data mart schedule stops being a bottleneck: model retraining is scheduled more often, new sources are connected without revising the architecture, and the failure of an individual node stops being an event people learn about in the morning. Next we will examine how such an environment is assembled and what remains on the customer’s side.
Spark job launch modes: batch, streaming, interactive
The launch mode in Apache Spark MLlib is chosen not by preference, but by the requirement for data delivery latency. Batch processing runs on a launch schedule: model retraining on a historical snapshot, bulk prediction calculation across a data mart, reporting reconciliation.
Streaming keeps the job running continuously and processes events as they arrive. Interactive queries are needed when an analyst selects features or hyperparameters and expects a response in seconds, not minutes.
The difference then shows up in resources and in how verifiable the result is. A batch job can be rerun on the same snapshot and produce identical output — hence reproducibility and clear rollback.
Streaming always carries state and checkpoints, so fixing an error retroactively is harder, and downtime costs more. Interactive mode lives until the session ends: it provides no scheduling or guarantees for production operation.
| Mode | When suitable | Limitations |
|---|---|---|
| Batch processing | Model retraining on history, regular calculation across data marts, reconciliations | Output latency — from minutes to hours; result is visible only after job completion |
| Streaming processing | Calculation on events as they arrive, online features, reaction to deviations | Resources are occupied continuously; state and checkpoints; harder to test and rerun |
| Interactive queries | Data exploration, feature selection, model debugging by an analyst | Session-bound, time and memory limits per query, not designed for regular load |
In practice, modes are combined: interactive for model development, batch for scheduled retraining, streaming for calculation on live traffic.
Training code transfers between them almost unchanged; what changes is the surrounding harness: data source, launch trigger, and quality control.
We check compatibility with current extracts before migration — on identical snapshots we compare the metrics of the new and existing model.
Implementation stages: from data audit to handover of procedures
Before the first line of code, we establish where data comes from and what happens to it along the way. The stage order is the same for a one-off calculation and for a continuously running pipeline. Each stage ends with an artifact that can be verified.
-
Data source audit. We analyze primary sources: extracts, databases, event queues, external services. We record access mode, update frequency, volume growth, and the data owner on the customer side.
-
Data profiling. We calculate the actual share of missing values and duplicates by keys, outliers, discrepancies in reference data and date formats between systems. Here it is also visible which fields are missing for the target task.
-
Processing schema design. We choose layers, partitioning keys, full or incremental load, and stopping points where it is better for the pipeline to fail than continue on corrupted data.
-
Environment setup. Cluster, storage, scheduler, separate environments for development and production runs, metrics and log collection. We connect Apache Spark MLlib into an already configured environment, not the other way around.
-
Validation on historical data. We run the pipeline on past periods and compare it with the reference calculation. We resolve discrepancies to an agreed threshold, while measuring full run time and headroom in the scheduled run window.
-
Solution handover. Launch and restart procedures, schema description, instructions for connecting a new source, team training. Then — support: logic edits and reaction to input data changes.
Implementation artifacts: configurations, schemas, procedures, documentation
Implementation ends not with a report, but with a set of working artifacts that remain in your environment.
Configurations, schemas, procedures, and documentation are in your repository under version control — so that an Apache Spark MLlib job can be brought up and maintained by your team without the author’s involvement.
- Job configuration: launch parameters, partitioning by data partitions, memory and parallelism limits, save points for intermediate results after heavy stages.
- Data flow schema: sources, normalization points, retraining step, result sinks — with formats and boundary conditions at each transition.
- Data access policies: roles and visibility scopes, masking of sensitive fields, grounds for extraction, and the procedure for granting rights to new employees.
- Restart procedure: conditions for automatic retry, signs of partially processed data, order of idempotent writes so that a retry does not create duplicates.
- Incident investigation instructions: degradation metrics, logging points, typical failures, and escalation procedure with responsible persons.
- Processing documentation: stage descriptions, field and feature dictionary, model limitations, and known boundary situations.
- Acceptance data and checks set: a reproducible run showing that changes did not break quality.
The check is simple: an engineer who was not involved in the implementation brings up the job from the documentation and gets the same result. If the author is needed to launch it, the kit is incomplete.
Case: processing volume tripled without rewriting the pipeline
Before restructuring, the scheme relied on one nightly recalculation: the pipeline launched entirely over all historical data and often failed to close the window.
Every fifth run failed on key skew and insufficient memory on executors, after which the calculation was restarted manually — data arrived partially by morning.
What changed
We kept Apache Spark MLlib as the computation core and restructured the jobs: the historical layer is calculated once and stored in columnar format, while daily windows reread only the last day’s delta.
Heavy aggregations were moved to a separate stage with sensible partitioning by natural key — skew disappeared, and stages stopped waiting for the most loaded partition.
For trained models, we added saving of intermediate states: a failure at a late step no longer means recalculation from scratch.
What became
The volume of processed data tripled in the same environment — without hardware replacement and without rewriting the calculation logic. Calculation time decreased from 6–8 hours to 40–50 minutes, the window closes with headroom, not by the time of the next run.
The share of unsuccessful runs dropped from every fifth to less than one percent, reruns are isolated, and almost all are caused by external sources, not calculation.
We separately introduced observability: for each step you can see how many rows were processed, where waiting grows, and at which stage skew increased. This allowed handling volume growth without proportional growth in calculation time.
How batch processing differs from streaming and what to choose
The choice of mode is dictated not by data volume, but by the freshness requirement: how long a decision can wait before the number becomes stale. Therefore, the first question at the start is not “how many events do we have,” but “what will change if data arrives in an hour.”
A batch scheme reads the accumulated layer for an interval and recalculates the result entirely: window aggregates, reporting data marts, model training.
Recalculation is reproducible — if there is an error in logic, the interval is relaunched and produces the same number, so rollback reduces to a rerun, and a failure only shifts the time when the report is ready.
A streaming scheme is justified where an hour’s pause breaks the scenario: reaction to an anomaly, stopping a suspicious operation, recalculating a metric in front of the user.
It brings computation state, late events, duplicate delivery, and the requirement for idempotent writes — maintenance complexity grows many times, and this is the price for the very ability to react in time.
In practice, schemes almost always coexist: streaming handles urgent signals, batch handles reporting, reconciliation, and retraining.
In Apache Spark MLlib, training runs on batch snapshots, while the streaming scheme applies the ready model to events as they arrive — so one feature calculation logic serves both modes, and the figures in the report match the fresh metric.
The practical test is simple: if an hour’s delay changes no decision, choose the batch scheme — it is simpler to develop and more predictable to maintain.
If the scenario loses meaning without fresh data, start with streaming, but immediately include latency monitoring and late event handling.
Technical limitations and common doubts before starting
Before planning work, we establish the boundaries of the approach. Apache Spark MLlib covers distributed training on large datasets, but result quality depends on data and the correctness of schemas, not on the number of executors.
Source data quality is the first constraint: missing values, duplicates, distribution shift between snapshots.
If the share of defects in the training sample is measured in percentages, metrics on the holdout set diverge from behavior on the real stream — and hyperparameter tuning does not fix this.
What happens if the source schema changes?
Schema backward compatibility is mandatory. Reading is done by explicitly declared schema, not automatic inference: a new column is ignored until explicitly connected, while renaming or a type change stops the pipeline. This is caught by a contract check at the input, before training starts.
Will the current environment be enough?
Distributed storage and a cluster that scales by memory and number of executors are required. The bottleneck is usually not computation volume but key skew: one key collects a disproportionate share of records, and some tasks run many times longer than others. It is addressed by repartitioning and pre-aggregation before training.
How do we make sure the model works?
The result is compared with a baseline on the holdout set and separately on slices — by segments and by time. Metrics are recorded together with the data version and launch parameters, otherwise the rerun is not reproducible.
A drop on one segment with good overall quality is a signal to return to the data, not to the model.
Let’s discuss the task: we’ll analyze the data and come back with a work plan
The start does not require a ready technical specification — access to data and an understanding of which result is considered successful are enough.
The sooner we see real extracts, the more accurate the timeline estimate: on synthetic examples the model behaves differently than on a live stream with missing values and duplicates.
What we need from you:
- a contact person authorized to approve the scope of work and priorities between stages;
- access to sources — database, extracts, logs, external services, with a description of who fills them and how;
- sample data for a typical period, including “dirty” areas: missing values, duplicates, different date and encoding formats;
- the result metric by which we will calculate the effect — share of correct predictions, response time, volume of manual verification;
- constraints: where processing is allowed to be deployed, requirements for storing personal data, who grants access;
- timeline priorities — what is needed for the first iteration and what can be postponed without losing meaning.
In response, we send source analysis, a processing schema on Apache Spark MLlib with a description of data preparation stages, the scope of work by stages, and a timeline estimate with the assumptions it depends on.
Then we agree on the first stage and start — usually this is loading, cleaning, and validation on a control sample, where it is visible whether data quality is sufficient for training.
Describe the task in two or three paragraphs — we will come back with an estimate and work plan.
