You rolled out a new version, and within half an hour — an avalanche of errors and a drop in conversion? Canary deployment prevents such scenarios: the new version gradually receives real traffic, and if metrics degrade, it automatically rolls back. We implement these schemes turnkey — from a simple Nginx config to automated rollout with Prometheus. Over years of practice, we have conducted more than 50 successful releases with canary. Our experience shows: without canary deployment, the risk of failure during a release increases fivefold.
Canary deployment — gradual traffic shifting to a new version: first 1–5% of users, then 10%, 25%, 50%, and finally 100%. It allows detecting issues on real traffic before full transition. Importantly, we guarantee that when the error threshold is exceeded (usually >1%), rollback happens in seconds. This approach provides three advantages: reduction of MTTR from hours to minutes, the ability to A/B test directly in production, and complete absence of downtime. Compare: with blue-green deployment, you maintain two full environments, while canary requires 30% fewer resources. According to the definition, Canary deployment is a rollout strategy where a new version of an application gradually receives real traffic (Wikipedia).
What Problems Does Canary Deployment Solve?
- Early detection of regressions on a small traffic share: errors that unit tests missed will appear on 1% of users, not all.
- Instant rollback without full redeployment: just change the weight to 0% — and users return to the old version.
- Testing new features on real users without staging costs: canary can be targeted to specific groups (by cookie or geo).
For example, on one project we discovered that a new API version caused an N+1 query to the database — on 5% of traffic latency increased by 200%. Canary automatically rolled back the version, and we fixed the issue without a mass outage.
Manual weight change in Nginx means 5 minutes of downtime to edit the config and reload. Automated rollout with metric checks reduces this to zero. We implement a pipeline that decides itself: increase weight or rollback. In Kubernetes with NGINX Ingress, rollback is 10 times faster — just delete the canary ingress.
Implementation: From Nginx to Kubernetes and AWS
Traffic is distributed between stable and canary versions: 95% of requests go to the old version, 5% to the new one. Load balancer or proxy determines which upstream to use.
Nginx split_clients
# /etc/nginx/nginx.conf
split_clients "${remote_addr}${http_user_agent}" $upstream_pool {
5% canary; # 5% → new version
* stable; # 95% → old version
}
upstream stable {
server 10.0.0.10:8080;
}
upstream canary {
server 10.0.0.11:8080; # new version
}
server {
location / {
proxy_pass http://$upstream_pool;
}
}
To change the percentage — edit the config and reload Nginx: nginx -s reload.
Canary via Cookie (sticky routing)
# The user always hits the same version
map $cookie_canary $upstream_canary {
"1" canary;
default stable;
}
# Or force enable for testers
map $http_x_canary_override $upstream_override {
"true" canary;
default $upstream_canary;
}
How to Set Up Canary Deployment in Kubernetes?
# stable-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp-stable
spec:
replicas: 10
selector:
matchLabels:
app: myapp
version: stable
template:
metadata:
labels:
app: myapp
version: stable
spec:
containers:
- name: myapp
image: registry/myapp:v1.0.0
---
# canary-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp-canary
spec:
replicas: 2
selector:
matchLabels:
app: myapp
version: canary
template:
metadata:
labels:
app: myapp
version: canary
spec:
containers:
- name: myapp
image: registry/myapp:v1.1.0
---
# canary-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: myapp-canary
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "5" # 5% traffic
spec:
rules:
- host: example.com
http:
paths:
- path: /
backend:
service:
name: myapp-canary-svc
port: { number: 80 }
Manage weight via kubectl: kubectl annotate ingress myapp-canary nginx.ingress.kubernetes.io/canary-weight=25 --overwrite. When fully transitioning, update the stable deployment and delete canary: kubectl delete ingress myapp-canary and kubectl delete deployment myapp-canary.
AWS: Weighted Target Groups
import boto3
elbv2 = boto3.client('elbv2')
def set_canary_weight(listener_arn: str, stable_tg: str, canary_tg: str, canary_weight: int):
"""stable_weight + canary_weight must sum to 100"""
stable_weight = 100 - canary_weight
elbv2.modify_listener(
ListenerArn=listener_arn,
DefaultActions=[{
'Type': 'forward',
'ForwardConfig': {
'TargetGroups': [
{'TargetGroupArn': stable_tg, 'Weight': stable_weight},
{'TargetGroupArn': canary_tg, 'Weight': canary_weight},
],
'TargetGroupStickinessConfig': {
'Enabled': True,
'DurationSeconds': 3600, # stickiness 1 hour
}
}
}]
)
Automated Canary with Metric Analysis
# canary-rollout.py
import time
import boto3
import requests
PROMETHEUS_URL = "http://prometheus:9090"
def get_error_rate(version: str, duration: str = "5m") -> float:
query = f'rate(http_requests_total{{version="{version}",status=~"5.."}}[{duration}]) / rate(http_requests_total{{version="{version}"}}[{duration}])'
r = requests.get(f"{PROMETHEUS_URL}/api/v1/query", params={"query": query})
result = r.json()["data"]["result"]
return float(result[0]["value"][1]) if result else 0.0
def progressive_rollout():
steps = [5, 10, 25, 50, 75, 100]
canary_weight = 0
for target_weight in steps:
print(f"Setting canary weight to {target_weight}%")
set_canary_weight(LISTENER_ARN, STABLE_TG, CANARY_TG, target_weight)
# Wait and check metrics
time.sleep(300) # 5 minutes per step
error_rate = get_error_rate("canary")
print(f"Canary error rate: {error_rate:.2%}")
if error_rate > 0.01: # >1% errors
print(f"Error rate too high ({error_rate:.2%}), rolling back!")
set_canary_weight(LISTENER_ARN, STABLE_TG, CANARY_TG, 0)
return False
print("Canary rollout complete!")
return True
How Does Canary Deployment Implementation Happen?
- Analysis of current architecture and traffic (1–2 days).
- Designing the canary scheme: choose tool (Nginx, K8s Ingress, AWS ALB) (1 day).
- Configuration setup and deployment (2–4 days).
- Integration with monitoring (Prometheus, Grafana, Datadog) (1–2 days).
- Testing and team training (1–2 days).
- Deployment and support during first days (1 day).
Total: from 5 to 10 business days depending on complexity.
| Method | Complexity | Implementation Time | Scaling | Rollback |
|---|---|---|---|---|
| Nginx split_clients | Low | 1–2 days | Limited | Manual (5 min) |
| K8s NGINX Ingress | Medium | 2–4 days | Automatic | Automatic |
| AWS ALB + Lambda | High | 3–5 days | Automatic | Automatic |
Monitoring Checklist for Canary Deployment
- Error rate on new version < 1%
- Latency p95 increased no more than 10%
- Conversion rate not decreased (if applicable)
- CPU/Memory within norms
- All external API integrations working
What Is Included in the Work
| What Is Included | Description |
|---|---|
| Canary configuration (Nginx/K8s/AWS) | Ready scheme with documentation |
| Monitoring and alerts | Prometheus + Grafana dashboards |
| CI/CD integration | GitHub Actions, GitLab CI, or Jenkins |
| Team training (1 session) | How to manage canary manually |
| Technical support for 2 weeks | Assistance during launch |
Over 7 years of experience, 50+ projects — our team is certified and ready to take on your project. Contact us for a consultation — we will assess your project in one day. Order turnkey Canary Deployment setup and get zero-downtime releases.
Timelines
- Nginx canary on VPS: 1–2 days
- Kubernetes NGINX Ingress canary: 2–3 days
- Automated rollout with metrics: 3–5 days