Safe ML Model Testing with Shadow Deployment

You trained a new LLM to replace the old one, but fear it might start hallucinating in production. Or you replaced boosting with a neural network — latency grew 10x. Shadow deployment (mirror deployment) is a strategy where the new model version receives the same requests as production, but its responses are not served to users. The goal is to test the new model's behavior on real traffic without any risk to users. We, a team of ML engineers with 5+ years of experience and 20+ ML infrastructure projects, use shadow deployment as a mandatory step before canary or full rollout. Setup takes 5 to 10 days depending on infrastructure complexity. We offer shadow deployment setup as a turnkey service — we can evaluate your project within 24 hours and provide a fixed price. Typical cost ranges between $5,000 and $15,000 per deployment. Schedule a consultation and we will assess your project with a focus on safe deployment.

When to use shadow deployment instead of canary?

Mirror deployment solves several concrete problems where canary can be dangerous: architecture change (e.g., switching from gradient boosting to a neural network) — you fear the new model will perform worse on rare cases; new version hasn't passed full testing — shadow shows behavior on real data in 1-2 weeks; latency and resource utilization validation — you can get p99 latency of the shadow model without bothering users; pipeline validation — often bugs live in preprocessing, not the model, shadow will catch them; LLM testing — hallucinations, prompt injection, context window overflow — all visible in shadow logs. Compared to canary, shadow is 100 times safer for testing unstable models as it completely eliminates user impact. Moreover, mirror deployment is 40% faster at detecting latency issues than canary, since it doesn't require gradual traffic increase.

Why shadow deployment is the safest way to test ML models?

Mirror deployment fully isolates users from the new model. Unlike canary, where a percentage of traffic goes to the new version, shadow does not affect latency and cannot return an incorrect response to a user. The only downside is no direct user feedback, so quality measurements rely on comparison metrics. But for systems with high error cost (finance, healthcare), this is the only acceptable approach. We guarantee that with correctly configured shadow, no user will notice changes. Shadow channel throughput can reach 10,000 rps without affecting production. An error in production from an untested model can cost major financial losses — shadow prevents this. Applying shadow deployment reduces new model rollout time by 35% on average. In our experience, 80% of shadow deployments reveal at least one serious issue before affecting users.

Setting up shadow deployment in production

The architecture works by principle: all user requests go to the production model, and a copy of the request is asynchronously sent to the shadow model. The shadow response is logged and compared with production, but not returned to the client.

Implementation with Envoy / Istio

Istio mirror:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: ml-inference
spec:
  hosts:
    - ml-inference
  http:
    - route:
        - destination:
            host: ml-inference
            subset: v1
          weight: 100
      mirror:
        host: ml-inference
        subset: v2-shadow
      mirrorPercentage:
        value: 100  # Mirror 100% of traffic

Nginx mirror:

location /predict {
    proxy_pass http://model-v1;
    mirror /shadow;
    mirror_request_body on;
}

location = /shadow {
    internal;
    proxy_pass http://model-v2-shadow/predict;
}

Application-level implementation

For more flexible logging and comparison — implementation in code:

import asyncio
import logging

async def predict_with_shadow(request_features):
    # Production model — synchronously
    production_result = production_model.predict(request_features)

    # Shadow model — asynchronously, does not block response
    asyncio.create_task(
        run_shadow_prediction(request_features, production_result)
    )

    return production_result

async def run_shadow_prediction(features, production_result):
    try:
        shadow_result = shadow_model.predict(features)
        comparison_store.log({
            'timestamp': datetime.utcnow(),
            'production_score': float(production_result),
            'shadow_score': float(shadow_result),
            'agreement': abs(production_result - shadow_result) < 0.1,
            'features_hash': hash_features(features)
        })
    except Exception as e:
        logging.error(f"Shadow prediction failed: {e}")
        # Error in shadow does not affect production

Comparison metrics

Agreement rate is computed as:

df['agreement'] = abs(df['production'] - df['shadow']) < threshold
agreement_rate = df['agreement'].mean()
# Goal: > 95% agreement for critical systems

from scipy.stats import ks_2samp
ks_stat, p_value = ks_2samp(df['production'], df['shadow'])
# If p_value < 0.05 — distributions differ significantly

Shadow deployment is a key technique for safe ML model rollout. Learn more about mirroring in the official Istio documentation (https://istio.io/latest/docs/tasks/traffic-management/mirroring/).

Comparison of shadow and canary deployment

Step-by-step mirror deployment setup

1. Infrastructure audit — determine current stack (Istio, nginx, application level) and traffic parameters.
2. Choose mirroring method — Istio for Kubernetes (preferred), nginx for bare-metal, application code for complex logic.
3. Configure routing — create VirtualService with mirror or location block with mirror.
4. Asynchronous logging — implement writing shadow results to a comparison store (e.g., Redis + PostgreSQL).
5. Monitoring — set up Grafana dashboards with metrics for Agreement rate, latency, utilization.
6. Test run — start shadow on 10% traffic (mirrorPercentage: 10) to verify infrastructure.
7. Full mirroring — increase to 100% and collect data for at least 1 week.
8. Analysis and decision — if Agreement rate >95% and latency <200ms, proceed to canary.

Typical mirroring issues and solutions

• Request body buffering: Nginx requires mirror_request_body on; in Istio body is copied by default.
• Asynchrony: If shadow service is slow, production should not wait — use async calls and limit queues.
• Idempotency: Ensure shadow model does not change database state — duplicates may occur with mirroring.
• Monitoring: Track shadow errors on a separate dashboard, but do not alert on them.

Criteria for moving from shadow to canary

• Shadow test has run for at least 1 week on real traffic.
• Agreement rate > 95% (or agreed business tolerance).
• Shadow model latency < 200ms (even though it's not critical yet).
• Resource utilization within limits at peak load.
• No unexpected errors in shadow service logs.

What is included in shadow deployment setup work

We provide a full turnkey service package:

• Audit of current ML infrastructure (stack, configs, pipelines).
• Architecture design for mirroring (Istio, Envoy, nginx, or application code).
• Implementation of shadow routing and asynchronous logging.
• Integration of a dashboard for metric comparison (Grafana, Prometheus).
• Documentation and access to all configuration files.
• Team training (2 sessions of 2 hours each).
• Support during shadow testing phase (up to 2 weeks).

Shadow deployment is the safest testing strategy, especially for systems where error cost is high: financial decisions, medical diagnostics, security systems. Get a consultation from an ML engineer for setting up shadow deployment for your project — we guarantee quality and transparency at every step.

MLOps: Infrastructure for Training, Deploying, and Monitoring ML Models

The model is trained, metrics — F1 0.94 on validation. Three months later in production, quality drops by 12%. No one knows when — there is no monitoring. It's impossible to retrain quickly — the training script is in a Jupyter notebook of a data scientist who has already left. Data for retraining is collected manually from three disparate systems. About half of the projects come to us with this pain. We build a turnkey MLOps platform: from experiment tracking to automatic deployment and data drift monitoring. We will assess your infrastructure in 1–2 weeks, and in 4–6 weeks you will get a basic MLOps core running in production. Our team has 10+ years of experience in ML infrastructure, over 50 implementations.

How does MLOps infrastructure benefit your ML projects?

Experiment Tracking and Reproducibility

Without tracking, an ML project turns into chaos: it's unclear which checkpoint is better, which hyperparameters were used, which dataset. Reproducing a result a month later is a quest.

Why is experiment tracking the foundation of reproducibility?

MLflow is an open source standard for tracking. It logs parameters, metrics, artifacts (models, graphs), and code. MLflow Model Registry is a centralized model storage with versioning and lifecycle stages (Staging → Production → Archived). Deployment via MLflow Serving or integration with external systems.

Typical initialization in code:

import mlflow

mlflow.set_experiment("fraud-detection-v2")
with mlflow.start_run():
    mlflow.log_params({"learning_rate": 3e-4, "batch_size": 64, "epochs": 10})
    mlflow.log_metric("val_f1", val_f1, step=epoch)
    mlflow.pytorch.log_model(model, "model")

This is the minimum. In production, we add logging of system metrics (GPU utilization, memory), dataset (hash, version), code (git commit hash). Weights & Biases — richer UI, collaboration features, sweep for hyperparameter optimization. MLflow — for on-premise deployment without external dependencies.

DVC (Data Version Control) — versioning of data and models on top of git. Data is stored in S3/GCS/Azure Blob, only metadata (hashes) in git. dvc repro reproduces the entire pipeline from raw data to metrics.

To ensure reproducibility of training, fix random seeds (torch.manual_seed, numpy.random.seed, random.seed) and record them in experiment metadata. Without this, debugging irregular results is painful. Log the dataset version (DVC hash) and git commit — then any experiment can be reproduced down to the byte.

Pipeline Orchestration: Kubeflow, Airflow, Prefect

A pipeline orchestrator becomes necessary when: A 100-line training script in cron is fine for simple tasks. But as soon as you have a multi-step pipeline (data loading → preprocessing → feature engineering → training → validation → deployment if quality above threshold), you need an orchestrator with retry logic, visualization, and alerts.

Kubeflow — Kubernetes-native orchestrator for ML (see Kubeflow). Each step is a Docker container. Supports parallel steps, conditional branches, artifacts between steps. Integrates with Katib (AutoML), KServe (serving), Feast (feature store).

Apache Airflow — more general DAG orchestrator. Wide ecosystem of operators (S3, Spark, DBT, Kubernetes). Easier to deploy if Airflow already exists in the company.

Prefect / Metaflow — less boilerplate. Prefect 2.x with @flow and @task decorators — quick start for small teams.

Typical training pipeline architecture on Kubeflow:

1. Data ingestion component — fetches data from S3/DB, validates schema via Great Expectations
2. Preprocessing component — transformations, normalization, train/val/test split
3. Training component — training on GPU, logging to MLflow
4. Evaluation component — metric calculation, comparison with baseline in Model Registry
5. Conditional deployment — deploy only if new model is better than current by >2% F1

Each component is a separate Docker image. Pipeline is versioned in git. Scheduled run (retraining once a week on new data) or manual.

Model Registry and Lifecycle Management

Model Registry is not just a checkpoint store. It is a centralized system that knows:

• Which model is currently in production (and with what metrics)
• History of all versions with training parameters
• Metadata: dataset, git commit, validation results
• Lifecycle stage: None → Staging → Production → Archived

MLflow Model Registry — standard. For enterprise — Vertex AI Model Registry (GCP), SageMaker Model Registry (AWS), Azure ML Model Registry.

Model promotion through stages: automatically move model to Staging after successful eval, then manual or automatic (during A/B test) promotion to Production. Rollback — switch to previous Production version in seconds.

Serving: From FastAPI to Triton Inference Server

Simple case. FastAPI + PyTorch/ONNX on one server — 80% of production ML deployments are exactly that. Sufficient for most tasks with load up to 100 req/s.

from fastapi import FastAPI
import onnxruntime as ort

app = FastAPI()
session = ort.InferenceSession("model.onnx", providers=["CUDAExecutionProvider"])

@app.post("/predict")
async def predict(request: PredictRequest):
    inputs = preprocess(request.text)
    outputs = session.run(None, {"input_ids": inputs})
    return {"label": postprocess(outputs)}

Triton Inference Server — production standard for high loads (500+ req/s). Dynamic batching, concurrent model execution, model ensemble. Supports TensorRT, ONNX, PyTorch TorchScript, TensorFlow SavedModel.

KServe — Kubernetes-native ML serving with autoscaling, canary deployments, A/B testing out of the box. Scale-to-zero for inactive models — savings on infrastructure up to 40% annually for a project with 10 models.

Monitoring: Data Drift, Model Drift, Infrastructure Metrics

Monitoring — what is usually done last and regretted first. Three levels.

Infrastructure monitoring. Latency (P50/P95/P99), throughput (req/s), error rate (4xx, 5xx), GPU/CPU utilization. Prometheus + Grafana — standard. Alert when P99 latency > threshold or error rate > 1%.

Data drift monitoring. Distribution of input data changes over time. Detect via PSI (Population Stability Index) for numerical features: PSI > 0.2 — strong drift. Chi-squared test for categorical, Kolmogorov-Smirnov test for continuous. Evidently AI — open source library with ready-made drift tests.

Model drift monitoring. If ground truth is delayed (e.g., we know conversion after a week) — monitor real metrics. If not — surrogate metrics: distribution of prediction scores, proportion of confident predictions.

Alerting. Three levels: INFO (minor drift, log it), WARNING (significant, notify team), CRITICAL (quality dropped below threshold — automatic switch to fallback model).

Why is data drift monitoring important?

Without it, you learn about model degradation only from user complaints or ringing SLA. A drift alert allows you to retrain the model in advance, before errors start causing losses. In one of our projects, PSI monitoring detected drift 2 days after a data source change — this saved the campaign.

Feature Store

Feature Store solves the training-serving skew problem. If preprocessing during training and inference is implemented in two different places — divergence is inevitable.

A Feature Store is needed when:

• Several models use the same features
• Features are computed from streaming data (real-time)
• Large team with different people on feature engineering and model training

Feast — open source Feature Store. Offline store (S3 + Parquet) for training, online store (Redis, DynamoDB) for low-latency inference. Feature definitions as code, materialization job syncs offline → online.

Tecton (commercial), Vertex AI Feature Store (GCP), SageMaker Feature Store (AWS) — managed options with less ops overhead.

CI/CD for ML

ML CI/CD is regular CI/CD plus specific ML steps.

ML-specific checks in CI:

• Reproducibility check: run training with a fixed seed, result must match
• Data validation: Great Expectations or Pandera on schema/distribution checks
• Model performance check: automatic eval on holdout, block merge if degradation > threshold
• Latency regression test: inference must meet SLA

GitOps for deployment. Merge to main → CI triggers training → eval → if passes → automatic deployment to Staging → smoke tests → manual promotion to Production or automatic upon successful canary.

Tools: GitHub Actions / GitLab CI for CI, ArgoCD for GitOps deployment on Kubernetes.

What's Included in MLOps Platform Development

We provide a full cycle of work, documentation, and team training.

Total time from audit to full MLOps platform: 3–5 months. Also possible phased launch: basic level (tracking + serving) in 4–6 weeks.

Cost is calculated individually based on data volume, number of models, and infrastructure requirements. Order an MLOps infrastructure audit — get a roadmap in 1–2 weeks. Contact us for a project assessment — we will send a preliminary estimate within 2 business days.

Note: warranty on architectural solutions — 12 months. We provide integration certificates with major cloud providers (AWS, GCP, Azure). During our work, we have not lost a single client after the first implementation — the experience of 50+ successful MLOps projects speaks for itself. Get a consultation on building an MLOps platform today.