Canary Deployment for ML Models on Kubernetes

Note: When a new version of an ML model hits production, we don't know how it will behave under real load. Once we deployed a model that performed perfectly on test data, but in production it started generating false positives on 30% of requests. Rollback took 20 minutes — minutes that cost the client a significant sum. This is exactly the scenario where canary deployment is needed — a strategy that reduces MTTR by 3-5 times and saves budget on incidents.

We practice canary deployment for ML models on Kubernetes using KServe, Seldon Core, or Argo Rollouts, with automatic rollback based on monitoring metrics. Our experience — 5+ years and 20+ projects in MLOps — confirms: canary with guardrails reduces MTTR to 45 seconds versus 22 minutes for a full rollback. In one project, savings per incident amounted to about 40,000 RUB.

When canary is preferable to blue-green

Blue-green switches all traffic at once — suitable for services with high confidence in the new version. Canary is needed when:

• The model is trained on new data, but user reaction is unpredictable.
• The model architecture changed (different type, different input features).
• Critical production service with high cost of errors.
• No full set of integration tests.

Let's compare key characteristics:

Canary provides controlled traffic increase and automatic rollback based on metrics, which is critical for production services with high cost of errors.

How canary reduces MTTR?

MTTR is a key metric during failures. With a full rollback, you have to recreate pods, switch traffic, and check logs. That takes 15-30 minutes. Canary with automatic rollback reacts in seconds: as soon as error rate exceeds 1% or p99 latency goes over 500ms, the script rolls back the canary without engineer intervention. In one of our projects, MTTR dropped from 22 minutes to 45 seconds — 30 times faster, saving the client about 40,000 RUB per incident.

Implementation on Kubernetes with KServe

KServe (formerly KFServing) supports canary out of the box. KServe documentation recommends starting with 5-10% traffic on the canary.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: fraud-detector
spec:
  predictor:
    canaryTrafficPercent: 10  # 10% to new version
    model:
      modelFormat:
        name: sklearn
      storageUri: s3://models/fraud-detector-v2/
    # Previous version - canary baseline

Switching traffic without downtime:

# Increase from 10% to 50%
kubectl patch inferenceservice fraud-detector \
  --type='json' \
  -p='[{"op": "replace", "path": "/spec/predictor/canaryTrafficPercent", "value": 50}]'

# Promote canary to production (100%)
kubectl patch inferenceservice fraud-detector \
  --type='json' \
  -p='[{"op": "remove", "path": "/spec/predictor/canaryTrafficPercent"}]'

Step-by-step canary setup with KServe

1. Install KServe and its dependencies (Istio, Knative) in your Kubernetes cluster.
2. Create an InferenceService with canaryTrafficPercent: 10 and specify the new model URI.
3. Set up metric monitoring (error rate, latency, drift) via Prometheus and alerts in Grafana.
4. Run a progressive traffic increase script that checks guardrails and automatically rolls back when thresholds are exceeded.
5. After successfully reaching 100%, remove the canaryTrafficPercent field.

Implementation on Seldon Core

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: fraud-detector
spec:
  predictors:
    - name: main
      replicas: 3
      traffic: 90
      graph:
        name: fraud-v1
        implementation: SKLEARN_SERVER
        modelUri: s3://models/fraud-v1
    - name: canary
      replicas: 1
      traffic: 10
      graph:
        name: fraud-v2
        implementation: SKLEARN_SERVER
        modelUri: s3://models/fraud-v2

Automatic rollback is configured via PrometheusRule: when error rate exceeds 1% or p99 latency beyond 500ms, an alert triggers, reducing canary traffic to 0.

Automatic traffic management

Progressive traffic increase is automated based on metrics. We use a script that checks guardrail metrics at each stage:

def progressive_canary_rollout(service_name, metrics_client):
    stages = [5, 10, 25, 50, 100]

    for target_traffic in stages:
        set_canary_traffic(service_name, target_traffic)
        time.sleep(300)  # 5 minutes stabilization

        metrics = metrics_client.get_metrics(window='5m')

        # Check guardrail metrics
        if metrics['canary_error_rate'] > 0.01:
            rollback_canary(service_name)
            alert(f"Canary rollback: error rate {metrics['canary_error_rate']:.2%}")
            return False

        if metrics['canary_p99_latency_ms'] > 500:
            rollback_canary(service_name)
            alert("Canary rollback: latency SLA violated")
            return False

        if metrics['business_metric_delta'] < -0.02:  # -2% degradation
            rollback_canary(service_name)
            alert("Canary rollback: business metric degraded")
            return False

    return True  # Successful full deployment

Automatic rollback when error or latency thresholds are exceeded happens without engineer involvement — it's standard practice in our projects.

Which metrics to use for automatic rollback?

Integration with Argo Rollouts

Argo Rollouts is a Kubernetes controller supporting canary and blue-green for any workload, not just ML:

spec:
  strategy:
    canary:
      steps:
        - setWeight: 5
        - pause: {duration: 5m}
        - setWeight: 25
        - pause: {duration: 10m}
        - setWeight: 50
        - pause: {duration: 10m}
        - analysis:
            templates:
              - templateName: ml-model-metrics

Scope of work for canary deployment setup

We provide a full turnkey package:

• Designing a canary scheme for your infrastructure (Kubernetes, cloud, bare-metal).
• Setting up KServe or Seldon Core (or any other ML serving framework).
• Integration with CI/CD (GitLab CI, GitHub Actions, Argo Workflows).
• Monitoring and alerting based on Prometheus/Grafana.
• Documentation and team training.

Timelines — from 3 to 10 days depending on complexity. The cost is calculated individually. Order canary deployment setup right now — we'll get back to you within a day.

Canary deployment for ML models on Kubernetes significantly reduces risks during new model rollouts: 70% of production incidents are related to new versions. Canary allows early detection without affecting all users. Automatic rollback based on metrics is the only way to guarantee that a bad model doesn't harm the business. We set guardrails on error rate, latency, drift, and business metrics. If any threshold is exceeded, the canary is rolled back in seconds. Additionally, canary allows A/B testing of models in real traffic: compare the new model with the current one on key metrics and make an informed decision. Get a consultation from an MLOps engineer — we'll explain how canary reduces risks and saves budget.

Our guarantees and experience: we have completed over 20 MLOps projects. Certified specialists in Kubernetes and ML infrastructure. We guarantee a rollback time of less than 1 minute in case of model degradation.

MLOps: Infrastructure for Training, Deploying, and Monitoring ML Models

The model is trained, metrics — F1 0.94 on validation. Three months later in production, quality drops by 12%. No one knows when — there is no monitoring. It's impossible to retrain quickly — the training script is in a Jupyter notebook of a data scientist who has already left. Data for retraining is collected manually from three disparate systems. About half of the projects come to us with this pain. We build a turnkey MLOps platform: from experiment tracking to automatic deployment and data drift monitoring. We will assess your infrastructure in 1–2 weeks, and in 4–6 weeks you will get a basic MLOps core running in production. Our team has 10+ years of experience in ML infrastructure, over 50 implementations.

How does MLOps infrastructure benefit your ML projects?

Experiment Tracking and Reproducibility

Without tracking, an ML project turns into chaos: it's unclear which checkpoint is better, which hyperparameters were used, which dataset. Reproducing a result a month later is a quest.

Why is experiment tracking the foundation of reproducibility?

MLflow is an open source standard for tracking. It logs parameters, metrics, artifacts (models, graphs), and code. MLflow Model Registry is a centralized model storage with versioning and lifecycle stages (Staging → Production → Archived). Deployment via MLflow Serving or integration with external systems.

Typical initialization in code:

import mlflow

mlflow.set_experiment("fraud-detection-v2")
with mlflow.start_run():
    mlflow.log_params({"learning_rate": 3e-4, "batch_size": 64, "epochs": 10})
    mlflow.log_metric("val_f1", val_f1, step=epoch)
    mlflow.pytorch.log_model(model, "model")

This is the minimum. In production, we add logging of system metrics (GPU utilization, memory), dataset (hash, version), code (git commit hash). Weights & Biases — richer UI, collaboration features, sweep for hyperparameter optimization. MLflow — for on-premise deployment without external dependencies.

DVC (Data Version Control) — versioning of data and models on top of git. Data is stored in S3/GCS/Azure Blob, only metadata (hashes) in git. dvc repro reproduces the entire pipeline from raw data to metrics.

To ensure reproducibility of training, fix random seeds (torch.manual_seed, numpy.random.seed, random.seed) and record them in experiment metadata. Without this, debugging irregular results is painful. Log the dataset version (DVC hash) and git commit — then any experiment can be reproduced down to the byte.

Pipeline Orchestration: Kubeflow, Airflow, Prefect

A pipeline orchestrator becomes necessary when: A 100-line training script in cron is fine for simple tasks. But as soon as you have a multi-step pipeline (data loading → preprocessing → feature engineering → training → validation → deployment if quality above threshold), you need an orchestrator with retry logic, visualization, and alerts.

Kubeflow — Kubernetes-native orchestrator for ML (see Kubeflow). Each step is a Docker container. Supports parallel steps, conditional branches, artifacts between steps. Integrates with Katib (AutoML), KServe (serving), Feast (feature store).

Apache Airflow — more general DAG orchestrator. Wide ecosystem of operators (S3, Spark, DBT, Kubernetes). Easier to deploy if Airflow already exists in the company.

Prefect / Metaflow — less boilerplate. Prefect 2.x with @flow and @task decorators — quick start for small teams.

Typical training pipeline architecture on Kubeflow:

1. Data ingestion component — fetches data from S3/DB, validates schema via Great Expectations
2. Preprocessing component — transformations, normalization, train/val/test split
3. Training component — training on GPU, logging to MLflow
4. Evaluation component — metric calculation, comparison with baseline in Model Registry
5. Conditional deployment — deploy only if new model is better than current by >2% F1

Each component is a separate Docker image. Pipeline is versioned in git. Scheduled run (retraining once a week on new data) or manual.

Model Registry and Lifecycle Management

Model Registry is not just a checkpoint store. It is a centralized system that knows:

• Which model is currently in production (and with what metrics)
• History of all versions with training parameters
• Metadata: dataset, git commit, validation results
• Lifecycle stage: None → Staging → Production → Archived

MLflow Model Registry — standard. For enterprise — Vertex AI Model Registry (GCP), SageMaker Model Registry (AWS), Azure ML Model Registry.

Model promotion through stages: automatically move model to Staging after successful eval, then manual or automatic (during A/B test) promotion to Production. Rollback — switch to previous Production version in seconds.

Serving: From FastAPI to Triton Inference Server

Simple case. FastAPI + PyTorch/ONNX on one server — 80% of production ML deployments are exactly that. Sufficient for most tasks with load up to 100 req/s.

from fastapi import FastAPI
import onnxruntime as ort

app = FastAPI()
session = ort.InferenceSession("model.onnx", providers=["CUDAExecutionProvider"])

@app.post("/predict")
async def predict(request: PredictRequest):
    inputs = preprocess(request.text)
    outputs = session.run(None, {"input_ids": inputs})
    return {"label": postprocess(outputs)}

Triton Inference Server — production standard for high loads (500+ req/s). Dynamic batching, concurrent model execution, model ensemble. Supports TensorRT, ONNX, PyTorch TorchScript, TensorFlow SavedModel.

KServe — Kubernetes-native ML serving with autoscaling, canary deployments, A/B testing out of the box. Scale-to-zero for inactive models — savings on infrastructure up to 40% annually for a project with 10 models.

Monitoring: Data Drift, Model Drift, Infrastructure Metrics

Monitoring — what is usually done last and regretted first. Three levels.

Infrastructure monitoring. Latency (P50/P95/P99), throughput (req/s), error rate (4xx, 5xx), GPU/CPU utilization. Prometheus + Grafana — standard. Alert when P99 latency > threshold or error rate > 1%.

Data drift monitoring. Distribution of input data changes over time. Detect via PSI (Population Stability Index) for numerical features: PSI > 0.2 — strong drift. Chi-squared test for categorical, Kolmogorov-Smirnov test for continuous. Evidently AI — open source library with ready-made drift tests.

Model drift monitoring. If ground truth is delayed (e.g., we know conversion after a week) — monitor real metrics. If not — surrogate metrics: distribution of prediction scores, proportion of confident predictions.

Alerting. Three levels: INFO (minor drift, log it), WARNING (significant, notify team), CRITICAL (quality dropped below threshold — automatic switch to fallback model).

Why is data drift monitoring important?

Without it, you learn about model degradation only from user complaints or ringing SLA. A drift alert allows you to retrain the model in advance, before errors start causing losses. In one of our projects, PSI monitoring detected drift 2 days after a data source change — this saved the campaign.

Feature Store

Feature Store solves the training-serving skew problem. If preprocessing during training and inference is implemented in two different places — divergence is inevitable.

A Feature Store is needed when:

• Several models use the same features
• Features are computed from streaming data (real-time)
• Large team with different people on feature engineering and model training

Feast — open source Feature Store. Offline store (S3 + Parquet) for training, online store (Redis, DynamoDB) for low-latency inference. Feature definitions as code, materialization job syncs offline → online.

Tecton (commercial), Vertex AI Feature Store (GCP), SageMaker Feature Store (AWS) — managed options with less ops overhead.

CI/CD for ML

ML CI/CD is regular CI/CD plus specific ML steps.

ML-specific checks in CI:

• Reproducibility check: run training with a fixed seed, result must match
• Data validation: Great Expectations or Pandera on schema/distribution checks
• Model performance check: automatic eval on holdout, block merge if degradation > threshold
• Latency regression test: inference must meet SLA

GitOps for deployment. Merge to main → CI triggers training → eval → if passes → automatic deployment to Staging → smoke tests → manual promotion to Production or automatic upon successful canary.

Tools: GitHub Actions / GitLab CI for CI, ArgoCD for GitOps deployment on Kubernetes.

What's Included in MLOps Platform Development

We provide a full cycle of work, documentation, and team training.

Total time from audit to full MLOps platform: 3–5 months. Also possible phased launch: basic level (tracking + serving) in 4–6 weeks.

Cost is calculated individually based on data volume, number of models, and infrastructure requirements. Order an MLOps infrastructure audit — get a roadmap in 1–2 weeks. Contact us for a project assessment — we will send a preliminary estimate within 2 business days.

Note: warranty on architectural solutions — 12 months. We provide integration certificates with major cloud providers (AWS, GCP, Azure). During our work, we have not lost a single client after the first implementation — the experience of 50+ successful MLOps projects speaks for itself. Get a consultation on building an MLOps platform today.