Kubeflow setup: why training is not reproducible in production
The model confidently shows the required quality in a notebook, but after rollout to the production environment the metric drops — and there is no way to explain the discrepancy.
The reason is usually not the algorithm: training ran on a different data extract, some steps were performed manually, and the environment differed from the production one. The team spends cycle after cycle on reruns and investigating “magic” instead of moving the model forward.
The failure consists of four things that live separately. Data is updated between runs, and the dataset version is not recorded anywhere. Feature preparation and training launch rely on manual commands, so a step is easy to skip or perform in a different order.
The mismatch between training and production environments finishes off the result: what worked on the test bench behaves differently in the production environment. And there is no experiment run log — there is nothing with which to compare two runs.
The cost of this state is not only in time. Reproducibility of model training stops being a property of the process and becomes luck: you cannot roll the model back to a verified state because it is unknown what it was assembled from. Incident analysis turns into archaeology through correspondence and script drafts.
This is solved by machine learning orchestration: data preparation, training, and validation are described as a single pipeline, run on a schedule or event, and record the inputs, parameters, and artifacts of each run.
Kubeflow provides a ready-made foundation for this, but it works only after careful setup — we break down how this is addressed.
What managed model training orchestration provides
Managed training orchestration on Kubeflow turns scattered scripts into a declarative pipeline: data preparation, training, quality validation, and artifact publication are connected into a single graph on the cluster.
A run is reproduced by a command rather than by a specific engineer’s memory, so the share of successful runs stops depending on who is on duty.
Model release time is reduced not by speeding up computations, but by eliminating downtime and blind reruns. A failure is localized to a specific stage instead of resetting the entire cycle, and training continues from the last successful step.
Reproducibility of runs and data versions
Each run records a link to the code version, a snapshot of the training set, and the resource profile, so during analysis it is clear what data produced the result. A quality discrepancy is explained by a specific change in input or configuration, not by guesses.
The class of “it works locally for me” errors, which cause models to be retrained blindly, goes away.
Fewer manual transfers between environments
A unified training environment eliminates copying weights and configurations between the experimental zone and the production environment. The same pipeline advances through stages with different resource limits, and credentials are injected from configured policies. The number of manual operations per release is reduced to approval and quality control.
In practice, this provides a predictable rhythm: the team runs experiments more often because a run costs attention, not time for manual environment assembly.
Launch options: from an engineer’s notebook to a managed cluster
The options differ not in “power,” but in who maintains the run state: an engineer manually, a pipeline on a schedule, or a cluster managed by an orchestrator.
This determines result reproducibility, the cost of downtime, and whether a colleague can repeat the run a month later without messaging the author.
| Option | When it fits | Limitations |
|---|---|---|
| Manual launch from a notebook | Data exploration, first hypotheses, one-off recalculations on a sample | No history of parameters and data versions, the result is not reproducible, the bottleneck is the engineer’s machine |
| Semi-automated pipeline | Regular training with manual step approval: data preparation, training, metric validation | A human remains a point of failure, steps are not isolated, and on failure the entire run is restarted |
| Managed cluster | Model flow, parallel experiments, requirements for logs and version rollback | Requires step descriptions, resources for each stage, and discipline in artifact versions |
| Hybrid scheme | Data cannot be taken outside, and training is heavier than local hardware | More complex networking and access, part of the logs lives on the customer’s side |
Where to run: cloud, own cluster, hybrid
The cloud provides a quick start and elastic resources for peaks, but runs into data storage policy. Your own cluster removes the access issue but requires workload-based loading and quota planning.
A hybrid puts data preparation and training in the customer’s environment and heavy computation outside; we pay for this with integration complexity and access rights analysis.
When the orchestrator is overkill
If a model is retrained once a quarter on one feature set, an orchestrator adds more maintenance than it saves: a described script and a run log are simpler.
Orchestration pays off when there are more than five steps, some of them repeat on a schedule, and the result must be reproducible by version number.
Setting up Kubeflow Pipelines: implementation stages
Kubeflow setup rarely comes down to installation — value appears when runs are reproducible and the training result is comparable with the previous one.
Therefore, we arrange the orchestration implementation stages sequentially: from assessment to handover, recording a verifiable result at each step.
Assessment of data, code, and current runs
- Run audit. We look at where training is manual, which experiments are not reproducible, and how much time is lost on repeating a step after a failure. The output is a prioritized list of bottlenecks.
- Data and code review. We check that data preparation, training, and model export are divided into independent steps with clear inputs and outputs. Otherwise the pipeline will be assembled, but the first failure will stop the entire process.
Agreement on the pipeline schema and environments
- Training pipeline schema. We draw the step graph: artifact versioning, restarting only the failed branch, and saving the parameters of each run. We agree on the schema before writing code — edits at this stage are cheaper.
- Environment separation. We separate development and production environments: identical settings, different rights and quotas. An experiment does not consume production training resources, and data access is granted by role.
- Validation on historical data. We run the pipeline on past datasets and compare metrics with the previous result: a discrepancy indicates an error in logic, not in the model.
- Handover and team training. We show how to add a step, roll back a version, and analyze a failed run. Documentation and the incident analysis procedure remain with the customer.
Handover artifacts: configurations, schemas, procedures
After project delivery, what remains is not a presentation, but a working set in your repository and internal knowledge base: configurations, schemas, procedures.
Everything is next to the code and versioned — a new engineer brings up the environment from documentation, not from the previous team’s memory.
Access policies and team roles
- Kubeflow orchestration manifests: all pipeline steps — data preparation, training, validation, model rollout.
- Data access policies: who launches experiments, who approves rollout, who sees training logs.
- Team roles and permissions: separation by namespaces, resource quotas, network rules between training and storage.
Model registry and data versioning
- Model registry with version history, quality metrics, and the “model → dataset → run configuration” link.
- Dataset versioning: sample snapshots, checksums, rules for reproducing any run.
- Procedures: how to roll back a model, how to release a new version, who is on duty when training fails.
- Team instructions: common errors, pre-rollout checklist, procedure for adding a new step to the pipeline.
- Architectural schemas and artifact naming convention so results are not lost between teams.
Metrics before and after: how the model release cycle changed
Case study: at the client’s team, model training was launched manually according to an instruction, and deployment to production was done by an on-duty engineer.
We broke down the path from data change to model update into versioned steps and moved orchestration to Kubeflow — with a run log, rerun from the failed step, and environment description as configuration rather than a set of “from memory” actions.
What we measured: cycle time and share of successful runs
We looked at three metrics: the share of training runs that completed successfully without manual intervention; the time from the appearance of new data to updating the model in production; the number of manual rollbacks per failed release.
Measurements were taken in the same way before the migration and one month after — on the same data stream, so the comparison was fair.
How the metrics changed after implementation
Before: about 60% of runs completed successfully; the rest failed due to environment version desynchronization and at the feature preparation step, which relied on a script without checks.
The model update cycle took an average of two to three weeks: the run and deployment waited for an available window from an engineer. Manual rollback was performed several times a month, and each took hours.
After: the share of successful runs rose above 95%, and failed runs are repeated automatically from the step where the failure occurred.
The cycle from data change to model in production was reduced to several hours — deployment goes through the same pipeline, not a separate release.
The class of failures due to environment version desynchronization disappeared: the environment is brought up from the same description as locally, so “it worked for me” stopped being an argument.
How does a pipeline rerun differ from retraining?
A pipeline rerun is the repeated execution of the same step graph on the same input data. Retraining a model is a new run of the training step on a changed sample, features, or settings.
These modes are often confused, and where minutes would suffice, an engineer waits for a full cycle.
During a rerun, the system compares the input fingerprint of each step: data composition, code, parameters, environment. If the fingerprint matches the previous run, the step is not executed — its output is simply pulled from the artifact storage.
This is how the intermediate result cache works: only those steps whose inputs actually changed are recalculated.
Incremental computations go further: a processing step can accept only the delta — new records for the period — and append it to the previous result without rereading the entire array.
This is correct where the logic does not depend on the overall context: window aggregates, conversion to precomputed constants.
Training starts over when what is considered the step’s input changes: sample splitting, feature set, data schema, model configuration. Here neither cache nor delta helps — a full run takes place, and it must be planned honestly, with a time buffer.
The speed of iterations depends exactly on this boundary. As long as edits concern preprocessing, metrics, or reports, the cycle takes minutes, and a dozen hypotheses are tested per day instead of one.
As soon as an edit affects the training sample, the cycle is again measured in hours — and this should be taken into account when planning experiments.
Compatibility, access, and data: technical concerns
Technical concerns about training orchestration usually come down to four: what to do with already written code, how to integrate the platform with the current data storage and model registry, how to separate teams, and what happens after handover. Below are short answers for each, with the level of detail needed for a decision.
Will existing training code have to be rewritten?
No. Your scripts are wrapped into container components: an image, input and output artifacts, and run parameters are described. The logic inside remains the same — only the way it is invoked changes.
Components are connected into a graph where the order of steps and data transfer are defined explicitly, not by a glue script.
Is this compatible with our data storage and model registry?
The platform is installed on top of S3-compatible object storage and works with network volumes, so data remains where it currently resides. The model registry is not replaced: the trained artifact is published to it via an API, and the version and metrics enter the run history. Data migration is not required — it is enough to grant workloads access to the necessary paths.
How are permissions separated and teams isolated?
Each team gets a separate namespace with its own quotas for CPUs, memory, and accelerators. Roles are described by access policies, service accounts are created for a specific pipeline — other teams’ artifacts and secrets are not visible. This removes the classic problem where one experiment consumes the resources of the entire cluster.
Can we roll back and reproduce a past run?
Yes. Not only models are versioned, but also run parameters, component images, and input datasets. Any past run can be repeated via a link to its configuration — this provides comparable results when analyzing quality degradation.
What happens after solution handover?
You receive operating instructions, deployment schemas, access policies, and a monitoring dashboard with alerts about failures and delays. Initially, we provide support: we analyze incidents together with your team, then transfer maintenance in stages. Support and response times are fixed in the contract in advance.
Describe your task — we will get back with an assessment and deployment plan.Let’s discuss the task and get back with an implementation plan
To come back with an implementation plan rather than general considerations, we need source data on how model training is currently organized.
Below is the minimum set: it already shows how many steps are being transferred to a managed form, where the bottlenecks are, and what cluster configuration will be needed.
- description of the current training process: which steps are performed manually, what is launched on a schedule, where data and model artifacts are stored;
- an example of one run — the set of steps, dependencies between them, quality checkpoints, and conditions under which the model proceeds further;
- team composition and roles: who prepares data, who trains, who puts the model into operation, and how many people work with the cluster simultaneously;
- environment isolation requirements: whether separate spaces are needed for teams and experiments, resource limits, and rules for access to external services;
- target training cycle time — how long the path from data change to a ready model currently takes and what time frame you consider acceptable;
- infrastructure constraints: where the cluster is deployed, security service requirements, storage, and networks.
With this set, we come back with an implementation plan: stages, scope of work, Kubeflow deployment scheme, roles on your team’s side, and checkpoints where the result is verified before the finish.
If some data is not yet available, we will discuss the task as it is and clarify what is missing in conversation. Describe the task — we will get back with an assessment and plan.
