Custom ML solution development: from prototype to production

Developing ML solutions: where pilots break

A pilot model on historical data almost always looks convincing: the sample is collected, the metric is calculated, the report matches.

Difficulties begin when the model is transferred into production — the same features arrive with a delay, some fields are empty, and user behavior has already shifted relative to the training sample.

The metric on the live stream drops, and the first to notice is not the analyst but the employee who manually corrects the result.

Training data quality is not a stage before launch but a permanent mode.

In historical extracts, the signal is cleaner than in the stream: events are pulled in late, duplicates are collapsed retroactively, and a feature that looked informative in labeling turns out to be a consequence of a past model decision rather than a cause.

That is why ML solution development starts with checking how each feature is formed at decision time, not at extract time.

Manual reviews are usually what eat up the effect. If an employee still reviews every second output, there is no gain in speed, and the cost of error remains the same.

We design the pipeline so that the model returns a decision with a confidence score: disputed cases go to a person, the rest is processed automatically, and the share of manual review is visible in monitoring.

Hence the readiness criterion for launch: the metric holds between retrainings, input data is checked at entry, and rollback to the previous version takes minutes. Next we break down what parts such a pipeline consists of and what is checked at each step.

What changes after the model is deployed into production

Before deployment, the model lives in a researcher's notebook: the result depends on who launched it and when, and any edit requires full manual reconciliation.

After moving to production, the model gets a version, an owner, and a rollback procedure — a model registry based on MLflow stores checkpoints and metrics for each build, so a new version is compared with the previous one on the same data, not "by eye".

Time to release a new version is reduced from weeks to several days: retraining, validation on a holdout set, and publication go through one pipeline.

The share of manual operations drops from 80% to 15–20% — only disputed cases remain manual, and they go into a queue for further training rather than being reviewed as a stream.

Failure classes also change. Previously these were "silent" errors: the model answered confidently and incorrectly, and they were learned about from user complaints.

Now for each class — discrepancy on a rare class, input data drift, unavailability of an external source — there is a counter, a threshold, and its own response scenario.

Load on reviewers stops growing with volume: the sample is labeled and checked by priority, not the entire stream.

Reproducibility of the result is fixed at the artifact level — the same input with the same version gives the same answer, and a discrepancy can be explained rather than attributed to "magic".

Solution delivery formats: service, environment, hybrid

Delivery format determines where model weights, training samples, and request logs live — infrastructure requirements and the level of control over the result depend on this.

The same algorithm can be delivered as a network service, deployed in the customer's private environment, or assembled hybrid: sensitive data is processed locally, while heavy computations go to dedicated capacity.

Choice is rarely dictated only by model quality. The fork is usually in three questions: can data be moved outside the perimeter, who owns the artifacts, and how often retraining on new data is planned.

What we clarify before choosing

We fix the format before development begins, together with data requirements and the model version rollback procedure. If conditions change, we move the solution between formats without rewriting the logic: the processing pipeline is separated from the model code.

Work stages: from data review to handover to the team

The order of steps is the same for forecasting models, document recognition, and generative assistants — only the data and acceptance criteria change.

Before assembly begins, we fix what we consider success: the share of correct answers on the holdout period, response time, and the share of manual corrections after the model.

1. Data review. We look at what is actually stored in accounting systems, extracts, and files: volume, labeling, gaps, duplicates, discrepancies in reference data. The output is a suitability report: which fields are sufficient, which features will have to be collected anew.

2. Solution design. We define the boundary between hard rules and the model, the response format for the consumer, and behavior for an uncertain result. We agree on the design before code — with a diagram and data flow description.

3. Build. We build a reproducible training pipeline: a single data source, fixed transformations, automated runs. Any result is reproduced on another machine — this is an acceptance condition.

4. Validation on historical data. We run the model on a period it has not seen and compare it with the current solution method. We separately calculate errors on rare cases — they most often undermine trust in the system.

5. Launch. We start with shadow mode: model outputs are calculated but do not change decisions. We compare discrepancies with specialists' work, then enable gradually and leave a rollback path to the previous processing order.

6. Handover and support. We hand over the retraining procedure, a set of checks, instructions for the shift, and metric control points. We agree on the update cadence on new data — otherwise quality drops in the first weeks after launch.

What you keep: code, designs, procedures, access

Solution handover is not "gave access to the repository and a farewell call", but a package by which your team raises, operates, and retrains the model without our participation.

That is why documentation and configurations are assembled during the project, not reconstructed from memory in the last week.

• Solution source code — data preparation and labeling, training, inference, launch scripts. A repository with change history, reproducible environment build, and pinned dependencies.
• Integration designs — how the model is embedded into your accounting and client systems: exchange points, request and response formats, timeouts, behavior when the source service is unavailable.
• Model documentation — a model card with a description of the training sample, metrics on the holdout set and by segments, known limitations, and signs of data drift.
• Access policies — roles and rights: who reads data, who launches retraining, who publishes a version to production. Logging of requests to the model and to sensitive fields.
• Support procedure — frequency of quality control, degradation thresholds, procedure for retraining and rollback to the previous version in one step.
• Deployment instructions — environment configurations, environment variables, data schema migrations, procedure for rolling out updates without downtime.
• Acceptance report — evaluation methodology, a test set of examples with expected answers, and thresholds below which a version is not released.

Project completion criterion is not a signed acceptance act, but a control run: your engineers on a clean environment assemble the environment, deploy the model, and repeat the quality measurement without prompts from our side.

If our engineer is needed at some step, then the artifact is incomplete, and we finish it before handover.

Case study: churn prediction for a service company

Before: monthly sample and manual decisions

The customer had a gap between data and action.

Once a month, an analyst exported history from billing and support tickets, merged it into one set, cleaned gaps, and assembled features manually — preparing the sample took three working days, and the risk report reached managers with a two-week delay. Priority was not backed by anything, so managers called the entire segment one after another.

The cost of such a scheme was working blind: up to 90% of calls did not end in renewal, and some clients had already submitted termination requests by the time of contact.

There was no way to check whether the retention logic itself worked — the monthly result was not compared with anything.

After: scheduled features and risk-based priority

Data preparation was moved into a regular pipeline: the feature store is recalculated on schedule in Airflow, with sources being billing, payment history, support tickets, and account activity.

The "churned / stayed" label is built on a sliding window, so the model trains under the same conditions in which it later operates. The same feature set that took three days to assemble is now updated in 40 minutes without analyst involvement.

Forecast is calculated every night, and the client card receives the risk level and the features that raised it — the manager sees the reason, not an abstract score: support contact frequency dropped, autopayment disappeared. 12–15% of the base goes into work instead of everyone in a row.

The share of manually made decisions decreased from 100% to 15% — the rest is covered by automatic renewal for low-risk clients. Precision of retention actions is 7 out of 10; churn in the target segment decreased by about a third over two quarters.

Quality is tracked by feature drift, retraining follows procedure, so the effect does not blur over time.

How do data for training and for model operation differ?

Training sample is a snapshot of the past: data collected, cleaned, labeled, and fixed. Live data stream behaves differently — it arrives continuously, with gaps, delays, and fields that change format without warning. A model trained on a snapshot encounters at launch what was not in it.

Hence the different behavior of the same features. In training, the value is known at decision time; in the stream, it arrives later — and the model decides on an outdated or empty field.

Then distribution shift: client behavior changes, seasonality, source composition, and relationships learned from the past stop describing the present.

We set feature control before launch, not after. We compare distributions of the training sample and live stream over sliding windows, keep a separate detector for each feature, and check that the model does not rely on fields that do not yet exist at decision time.

We separately catch leakage: if a feature in training was calculated from data unavailable in operation, offline metrics will be excellent, while in the stream the model will fall apart.

Pre-launch validation is a run on a time-based holdout period, not a random sample, with artificial delays and source failures.

Then shadow mode: the model calculates but does not affect decisions, and the discrepancy between forecast and fact is visible before it reaches the user. After launch, degradation thresholds, scheduled retraining, and rollback to the previous version remain.

Technical concerns of customers: data and quality

Before start, almost every customer asks the same questions: what to do with incomplete data, how to notice quality degradation, how labor-intensive model replacement is, and who stays nearby after launch.

Below are short engineering answers to them, without promising that everything will work perfectly on the first run.

How can we tell that quality has dropped?

We set up quality control on the stream: a reference set of examples, the share of erroneous answers, and tracking of input data shift. Metrics are calculated regularly, not after a user complaint, so an accuracy drop is visible before it reaches the business process. If the threshold is exceeded, retraining on fresh data is triggered.

What happens if the model has to be replaced?

The model is isolated behind a single interface, so replacement does not break integrations on the customer side. We change the algorithm, run the new variant on the reference set, and enable gradually, with the ability to return to the previous one. No need to rewrite accounting systems and interfaces.

Why is your team needed after launch?

We hand over documentation, retraining procedures, and the incident review process, and train your engineers. Support then covers what requires regular hands: data updates, quality control, point tuning of rules. The customer can at any moment bring support in-house — code and designs remain with them.

Let's discuss the task and come back with an estimate and plan

To come back with an estimate and plan, rather than general wording, we need context on the task from you — in the form in which it exists. An extract, a process description, and a person responsible for the result on your side are enough.

• Data: access to storage or an extract, examples of records, whether labeling exists and by what procedure it was done.
• Goal in measurable form: what we consider success — share of correct answers, recall, response time, share of manual corrections.
• Constraints: where the solution will operate, with which systems it will exchange data, what cannot be moved outside your environment.
• Current environment: what is already assembled — models, scripts, processing rules, accumulated labeling.
• Responsible people and access: who approves requirements, who accepts a stage, to whom access to data and environment should be opened.
• Expected load: how many records or requests per day, how many scenarios we cover in the first version.

Next we review requirements together with the engineers who will lead the project — without passing the task "through an intermediary".

As a result, you receive a document: scope of work by stage, timeline for each stage, what is required from your side, and which risks we see in advance. Describe the task in a convenient form — we will come back with an estimate and plan by which a decision can be made.