DR Drill: Testing Recovery Procedures and the Team

DR Drill: проверка процедур восстановления и команды

Companies regularly invest in backup and fault-tolerant infrastructure but often fail to verify whether it will work during a real failure. The first disaster recovery drill (DR Drill) usually reveals unpleasant surprises: a database backup takes 6 hours to restore instead of the planned 30 minutes, and the runbook contains commands for a server that has long been decommissioned. Such issues go unnoticed until an incident, when every minute of downtime costs tens of thousands of dollars. For example, in e-commerce, downtime during peak loads can lead to millions of rubles in lost revenue per hour. During a peak season failure, losses can exceed $100,000 per hour. We conduct comprehensive DR drills to identify weaknesses in team procedures before they lead to a catastrophe. With over 7 years of experience and more than 50 completed drills, we guarantee the detection of critical issues.

What problems does the first DR Drill uncover?

Unverified backups are an illusion of security. During the first drill, we often find:

• Backup exists, but restoration takes 6 hours instead of the expected 30 minutes
• Configuration files are stored only on the primary server and are not included in backups
• Secrets (API keys, certificates) are kept in people's heads rather than in a Vault/Secrets Manager
• The runbook describes outdated infrastructure
• The team does not know who makes the decision to activate DR

These problems directly affect RTO and RPO. According to a Disaster Recovery Journal survey, 53% of organizations fail to meet their stated RTO during actual recovery. Regular drills are the only way to verify whether your procedures match your promises. Drills save an average of $50,000 by identifying issues before an incident.

Why are regular drills critical?

Drills allow you to test not only the technical side but also communication processes. In a stressful situation, the team acts differently than on paper. Regular drills turn recovery into routine rather than a fire drill. They also help justify infrastructure budgets: after drills, you know exactly which components need reinforcement. For example, one client discovered that promoting a PostgreSQL replica took 4 times longer than the RTO — after the drill, we optimized the configuration and reduced it to 2 minutes. The savings from preventing a single incident can be up to $200,000.

How we prepare and conduct a DR Drill?

Our approach starts with an audit of the current infrastructure and documentation. We update the runbook, check backups, and assign roles. Then we select a scenario (see table below) and conduct the drill in an agreed-upon window. After the drill, we record actual metrics and create an improvement plan.

Comparison of drill types

Tabletop exercises are good for initial checks — they take only half a day and do not touch production. Functional exercises uncover issues with specific tools, such as errors in backup scripts. Full-scale drills are the most realistic but require careful preparation. In our practice, full-scale drills are 3 times more effective than tabletop exercises for testing communications.

How to conduct a successful DR Drill?

1. Update the runbook. Verify that all commands and server addresses match the current infrastructure.
2. Check backup freshness. Ensure the latest backup is complete and available for restoration.
3. Assign roles. Determine who makes decisions, who executes steps, and who records timestamps.
4. Choose a scenario. Start with tabletop, then progress to functional, and only then to full-scale.
5. Conduct a post-mortem. Compare actual metrics with expected RTO/RPO and create an improvement plan.

What is included in the work

We provide turnkey:

• Updated runbook with step-by-step instructions
• Drill report with time metrics and deviations
• Updated procedures and improvement recommendations
• Team training and documentation handover

According to the NIST SP 800-34 methodology (https://www.nist.gov/privacy-framework/nist-sp-800-34), regular drills are a mandatory part of a business continuity program.

Typical mistakes in organizing DR

• Conducting drills without a preliminary tabletop — jumping straight to full-scale without knowing if the runbook works.
• Not assigning an observer — without recording time, it's impossible to estimate RTO.
• Ignoring post-mortem — drills without analysis are just wasted time.
• Thinking one full-scale drill per year is enough — infrastructure changes in between, and functional exercises help stay in shape.

If you want to test your resilience to failures, contact us for a consultation. Preparation for the first tabletop exercise takes 2-3 days, organizing a functional exercise takes 3-5 days, and a full-scale drill takes 1-2 weeks. The cost is calculated individually based on infrastructure complexity and the chosen scenario. To get a consultation and estimate timelines for your project, contact us. To order a DR Drill, contact us and receive a professional analysis of your infrastructure.

We regularly encounter a situation: "The site is not opening" at 3 a.m. — and it turns out that the VPS disk is full because nginx logs haven't been rotated for six months. Or the server went down under load on the day of an advertising campaign launch because the shared hosting had a limit of 50 concurrent connections. Setting up hosting and deployment is not about "where it's cheaper" but about what happens when something goes wrong. Our team helps avoid such incidents by designing infrastructure that accounts for real load patterns.

When to choose Vercel and Netlify?

Vercel is built for Next.js — deploy in one push, preview deployments for every PR, automatic CDN, Edge Functions, ISR without configuration. For frontend projects and JAMstack, it's the optimal choice: no operational overhead, time-to-deploy measured in minutes.

Real limitations: Vercel Serverless Functions run in us-east-1 by default (latency for Europe +80–100ms), Function timeout 300 seconds on Pro, Bandwidth 1TB/month on Pro. For heavy backend, you need workers or a separate server.

Netlify is closer to static sites and Edge Functions based on Deno Deploy. Build minutes are the main limitation on the free tier.

Why is Docker the foundation of predictable deployment?

"It works on my machine" — classic. Docker solves this through environment containerization. But a bad Dockerfile creates new problems.

A typical mistake: copying everything into the image without .dockerignore, resulting in an 800MB image instead of 80MB. node_modules inside the image weighs as much. Correct approach: multi-stage build.

FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
RUN npm run build

FROM node:20-alpine AS runner
WORKDIR /app
COPY --from=builder /app/.next ./.next
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/package.json ./package.json
EXPOSE 3000
CMD ["npm", "start"]

Final image: 180MB instead of 1.2GB. CI build time is reduced due to layer caching — if package.json hasn't changed, the layer with npm ci is taken from cache.

Docker Compose for local development and simple production scenarios: application + PostgreSQL + Redis in one configuration. For production on a single server, it's a perfectly viable option if there's no requirement for horizontal scaling.

More about containerization — Wikipedia: Docker.

How to set up Nginx as a reverse proxy?

Nginx in front of the application is standard for VPS and dedicated servers. Main functions: SSL termination, gzip, static files, rate limiting, upstream load balancing.

A configuration often done incorrectly: worker_processes auto — number of processes equals CPU count. worker_connections 1024 — that's 1024 per worker process. With 4 CPUs and 1024 connections = 4096 concurrent connections. For a high-traffic site, you need worker_connections 4096 and set keepalive_timeout 65.

For static assets with hash in the filename:

location ~* \.(js|css|woff2|png|webp)$ {
    expires 1y;
    add_header Cache-Control "public, immutable";
}

immutable tells the browser: don't revalidate this file even on hard refresh. This only works correctly with content-hashed filenames (which Vite/webpack do by default). Documentation — Wikipedia: Nginx.

AWS: flexibility and complexity

EC2 + Auto Scaling Group — classic for horizontal scaling. AMI with pre-installed application, Launch Template, ASG with min/desired/max instances, Application Load Balancer. When CPU > 70% for 3 minutes — scale out, when CPU < 30% for 15 minutes — scale in. Health check via ALB removes unhealthy instances from rotation.

ECS Fargate — containers without managing EC2. Deploy a Docker image, specify CPU/memory (512 CPU units = 0.5 vCPU, from 512MB memory), Fargate launches it. More expensive than Lambda, but no cold start and no timeout limitations. Suitable for long-running processes, WebSocket servers, heavy workers.

RDS for PostgreSQL with Multi-AZ: automatic failover in 1–2 minutes when primary fails. Read Replicas for scaling reads. RDS Proxy for connection pooling — Lambda functions cannot hold long-term connections, the proxy buffers this.

Kubernetes: when it is justified

K8s adds significant operational complexity. Justified when: multiple teams deploy independent services, fine-grained resource allocation per service is needed, canary deployments and blue/green without downtime are required.

AWS EKS, GKE, or managed k8s from Hetzner (cheaper). Helm charts for standard services. Horizontal Pod Autoscaler based on CPU and custom metrics (RPS via Prometheus).

For most startups and medium-sized projects, Kubernetes is overkill. ECS or Fly.io provide 80% of the capabilities with 20% of the operational complexity.

Monitoring and alerting

A server without monitoring is waiting for an incident. Minimal stack: Prometheus + Grafana (or Grafana Cloud for managed), alerting on disk > 80%, memory > 85%, CPU > 90% over 5 minutes, error rate > 1%. Uptime via Better Uptime or Upptime (self-hosted).

Logs: Loki + Grafana or CloudWatch Logs Insights. Structured JSON logs (winston, pino) are mandatory — otherwise, log searching becomes a pain.

What is included in hosting setup

• Audit of current infrastructure and load profiling
• Selection of target architecture (VPS, AWS, serverless, Kubernetes)
• Setting up CI/CD pipeline (GitHub Actions, GitLab CI) with automatic deployment
• IaC via Terraform or Pulumi (infrastructure as code)
• Configuration of Nginx, SSL certificates, HTTP/2, brotli
• Monitoring and alerting (Prometheus + Grafana, PagerDuty)
• Documentation of runbooks and team training

Additionally, contact us if you need migration from current hosting or integration with external services.

Work process

1. Audit of current infrastructure (2–5 days)
2. Selection of target architecture with load and budget justification (1–3 days)
3. Setting up CI/CD pipeline (GitHub Actions, GitLab CI) (2–5 days)
4. IaC via Terraform or Pulumi (3–10 days)
5. Setting up monitoring and alerting (2–5 days)
6. Documentation of runbooks and team training (1–3 days)

Our experience — 7 years on the market, over 50 projects, guarantee of operability after deployment.

Timeline

• Basic deployment on VPS with Docker + Nginx + CI/CD: 1–2 weeks.
• Setting up AWS infrastructure with Auto Scaling, RDS, CDN: 3–6 weeks.
• Migration to EKS from scratch: 6–12 weeks.
• Setting up Vercel/Netlify for JAMstack: 3–5 days.

The cost is calculated individually depending on complexity and scope of work. Get a consultation — we'll evaluate your architecture in one day.