Setting Up A/B Testing of ML Models in Production

A/B testing ML: where production breaks

The most common picture: a new version of the model is rolled out to production entirely, in a single release, on top of the old one.

There is nothing to compare it with—the previous version no longer receives traffic, while offline evaluation on a holdout set shows one thing, whereas the live system operates by different rules.

The dispute over whether it has become better is settled not by measurement but by who insists more loudly.

The discrepancy in metrics almost always comes from sample bias. The holdout set is collected on past data, while live traffic changes: new segments arrive, the share of users with history changes, and rare requests behave differently.

Quality control of the model by a single average figure hides this—the metric grew overall, but declined on the target segment.

A viable scheme is fractional A/B testing of ML on live traffic: some requests go to the new version, some remain in the control. The user is assigned to a branch for the entire session; otherwise, comparison of model versions is blurred by switches.

Shadow mode and canary rollout provide the same measurement without risk: the shadow copy computes answers without affecting the output, while the canary share grows as metrics are confirmed.

Then the rollout of the model to production relies on measurement, not opinion: the gain on the target segment is visible, along with the cost—latency and failure rate. How this is addressed in practice is below.

What controlled model comparison on live traffic provides

The hypothesis testing cycle accelerates

Previously, a new version of the model waited for the nearest release, and the result was evaluated by indirect signs and complaints. Now traffic is divided into groups directly in the production loop, and the time to result evaluation is measured in days, not weeks. In the same period, the team manages to test more hypotheses and closes dead-end branches faster.

Comparing user groups instead of arguing

Both versions receive comparable request streams, and audience behavior is compared across pre-agreed slices.

This removes the argument "I think the new model answers better": the decision relies on the difference between groups, not on the impression from a few illustrative examples.

A unified quality metric

The scale is fixed before the start—the share of successfully completed scenarios, answer accuracy on the control set, depth of interaction.

One metric for all iterations makes it possible to compare versions with each other and see which step produced a gain and which created only the appearance of work.

The share of losing rollouts falls

While comparison is running on part of the audience, an error manifests before it affects everyone.

A weak version is removed from the stream and does not reach a full release—the class of failures where an "improvement" imperceptibly worsened quality leaves the process.

Rollout formats: shadow mode, canary, full A/B

The choice of rollout mode is primarily a choice of the cost of error. Shadow execution does not create it, canary release limits it to the share of traffic in the test, and a full A/B test puts the entire audience at stake.

The difference between formats is not in "advancedness" but in which constraints you are willing to accept: data availability, observation period, and rollback requirements.

In practice, the formats do not compete but form a chain: shadow execution filters out obvious failures, canary release checks stability under live load, and a full A/B test answers the question about uplift.

The key condition for moving to the next step is a unified set of metrics in all modes and a working rollback to the previous version in each of them. Otherwise, the results are incomparable, and the rollout decision will have to be made by eye.

How this looks on live traffic was discussed in Case Study of Rolling Out a Recommendation Model—there it is also clear why control on a share of traffic is still mandatory after shadow execution.

Implementation stages: from data audit to handover of the loop

Implementing A/B testing of models is not installing a library but a sequence where each step relies on the previous one. A skipped data audit leaves the experiment design without support, and the conclusions cannot be defended afterward. Our working order is as follows.

1. Data audit. We look at which events are logged, with what delay they arrive, where the share of outcomes is lost, and whether there is a historical slice for offline execution. Without this, it is impossible either to calculate the metric or to estimate the test duration.
2. Experiment design. We fix the primary metric, guardrail metrics, hypothesis, minimum detectable effect, and stopping criteria—before the start, not after the first look at the numbers.
3. Group splitting. We assign by user identifier, not by request: otherwise one person falls into both groups and the comparison is blurred. Before launch, we check group balance by history slice.
4. Loop setup. We configure routing of the traffic share and logging of the model version together with the decision. Every impression must be linked to an outcome; otherwise, analysis of discrepancies rests on guesses.
5. Pilot validation. We launch on a small share of traffic and look at technical errors, response time, and input data drift. We expand only after the loop behaves stably.
6. Loop handover. We provide the regulation for launching new versions, the dashboard, accesses, and the procedure for stopping the experiment. The client's team runs the next test itself.

Configurations, schemes, and experiment regulations

An experiment is manageable when everything that determines its behavior lies in files and regulations, not in an engineer's head.

Below is the minimum kit that remains with the client along with the working loop: from it, the run can be repeated, stopped, or rolled back without our team's involvement.

• Experiment configuration—traffic shares between control and variants, observation horizon, minimum sample size, early stopping criterion, and winner declaration rule.
• Metric map—one target metric with the direction of change, guardrail metrics that must not decline, and a breakdown by segments where the effect may differ.
• Traffic flow diagram—how a request gets into a variant, where the distribution is recorded, and who is responsible for ensuring that the same user remains in their group.
• Data access rules—who sees raw events and on what basis, where anonymized slices are stored, and how validation labeling is separated from the training set.
• Loop documentation—composition of services, dependencies, integration points with storage and monitoring, and the procedure for regular startup and shutdown.
• Rollback regulation—triggers (decline in a guardrail metric, increase in failures, distribution discrepancy), the responsible person, time to return to control, and the analysis procedure.
• Run log—hypothesis, model version, period, result, rollout decision, and its author.

Each item is a working artifact: it is read during shift handover and incident analysis, not only at acceptance. The kit is updated together with the loop—otherwise, after a couple of iterations, the documentation diverges from reality, and rollback turns into manual archaeology.

Case study: how we removed discrepancies between model versions

What the input was

The client was a service with a ranking model in production. The team rolled out a new version once every two weeks and each time got a disputed result: on one segment the model won, on another it lost, and the decision was made based on the average across all traffic.

The failure rate on new versions remained at 6–8%, and analyzing one hypothesis took about three weeks—the effect managed to become outdated before the test closed.

What we changed

We separated two different questions: "is the model actually better?" and "is the traffic divided fairly?" Stratification of the stream by key segments appeared, along with fixing the randomness of splitting, shadow execution of the new version on all traffic before part of the users see it, and guardrail metrics—response time and share of empty outputs—with automatic stopping of the test on decline.

The stopping criterion was set in advance, so the rollout decision stopped depending on who insists more loudly.

What became

The failure rate on new versions decreased from 6–8% to 2%, and the spread of the effect between segments became visible before rollout, not after. The time from hypothesis to decision shrank from three weeks to six days.

Stability was checked with repeated runs: the sign of the effect matched in nine out of ten runs, whereas previously the result changed from run to run.

The approach was not invented by us—group stratification and guardrail metrics are discussed in detail in Microsoft Research materials on building experimentation platforms; we adapted them to the client's scale and data structure.

How A/B testing ML differs from canary release

A/B testing ML answers the question "has it become better," while canary release answers "has it broken." The first compares model versions by a user metric; the second insures the rollout against regression.

A canary stage with error thresholds will let through a model that works stably but loses conversion.

The difference begins with the unit of observation. Canary release operates with service instances and traffic shares—aggregates over minutes are enough for it.

In A/B testing ML, the unit is a user or a session assigned to a group: if the same traffic goes to both branches, sample bias arises, and the model effect is not separated from the audience effect.

The decision based on the test result is also made differently. Canary release is closed by automation: exceeding failure and latency thresholds triggers a rollback; otherwise, the share is expanded.

A/B testing ML requires a pre-selected metric, power calculation, and observation window—usually weeks, to capture the weekly cycle of behavior.

Data constraints diverge: on a weak stream, A/B testing ML does not gain power, while canary release starts on a small share and does not accumulate statistics.

In practice, they are combined—first the canary stage, then A/B on the remaining audience; we keep the metrics of both stages in one experiment tracker, for example MLflow.

Questions on data, metrics, and test infrastructure

Doubts around tests repeat: can the difference between groups be trusted, what to do about labeling delay, and whether to deploy your own experimentation loop. Below are short answers to what most often comes up at the evaluation stage.

Labels arrive with a delay—can the test be trusted?

Yes, if the observation window is coordinated with the label maturation horizon. We count both groups in the same window and recalculate the result after it closes, not from the first days, where some events have not yet arrived.

Is a separate experimentation loop needed?

Usually not: if traffic splitting, impression logging, and metric calculation already work, the model version and group indicator are added to them. We deploy our own loop when decisions need to be made faster than current analytics allows.

How do we achieve comparability of groups?

We split by a stable user identifier, not by session or individual request, and before the start we compare groups by key slices. Otherwise, a difference in traffic composition is read as a model effect.

Let's discuss the task: we will put together an experiment plan for your stream

For the experiment plan to be realistic rather than generic, we need input data from your side.

It is enough to describe the current request stream, product metrics, and constraints—we will clarify the rest in correspondence, and we will address contentious points with hypotheses and mark them separately in the plan.

• How the decision is made now: one model for all traffic, manual rules, or a previous experiment—and what exactly did not work in it.
• What share of traffic can go to the test, and where the entry points are: an online request service, a queue, scheduled batch recalculation.
• Success metric in two layers: a business indicator (conversion, cancellation share, processing time) and the model's technical accuracy with spread across segments.
• Over what period the conclusion is needed—days or weeks—and whether the test can be extended if there are not enough events for a significant result.
• What is available in terms of data: logs of previous versions, a labeled sample, access to storage, and fields that cannot be exported externally.
• Rollback requirements: in what time the previous model must be restored and who makes the decision to stop the experiment.
• Who on the client side approves metrics and accesses—one responsible person or several teams.

In response, we will provide an estimate of the scope of work, the stage-by-stage work order, the traffic splitting scheme, and stopping and rollback criteria. Describe the task—we will put together a plan for your stream and provide timelines.