On-Call Rotation Setup for Support Teams

We encountered a situation: a three-person support team burned out within two months due to chaotic night calls. Incidents piled up, MTTA grew to an hour. The solution was implementing on-call rotation with clear escalation rules. Within two weeks, MTTA dropped to 5 minutes, and the load was evenly distributed. Proper configuration pays off within the first month, and the cost of organizing rotation varies depending on the volume of integrations.

Problem Analysis: Why Teams Burn Out on Duty

On-call rotation is a system where the responsibility for responding to incidents outside working hours is distributed among team members in turns. Without rotation—one on-call engineer burns out in a month. With correctly configured rotation—the load is even, and response is predictable. Our experience shows that weekly rotation is 3 times better than monthly in terms of context recovery speed.

Designing the Rotation Scheme

How to Choose a Rotation Period?

Primary on-call: first level, receives all alerts. Response time—5–15 minutes. Secondary on-call (backup): if primary doesn't respond within 10–15 minutes—escalation to secondary. Escalation path: Primary → Secondary → Engineering Manager → CTO. Each level adds 10–15 minutes.

On-Call Tools: PagerDuty and Alternatives

Setting Up Escalations in PagerDuty

Service → Escalation Policy:
  Level 1: On-Call schedule (primary)
    - Notify after: immediately
    - Escalate after: 15 minutes
  Level 2: On-Call schedule (secondary)
    - Notify after: escalation
    - Escalate after: 15 minutes
  Level 3: Engineering Manager
    - Notify after: escalation

Schedule (Primary):
  Rotation type: Weekly
  Handoff time: Monday 10:00 local time
  Restrictions: None (24/7 coverage)
  Layer 1: [Engineer A, Engineer B, Engineer C, Engineer D]

Handoff during working hours—the engineer takes over the shift in a calm environment, reviews open incidents.

Integration with Monitoring

Integrate monitoring (Prometheus, Datadog, Sentry) with PagerDuty via webhook. Configure smart notifications: Slack during the day, call at night. This reduces alert fatigue and improves MTTA.

Implementation: Step-by-Step Guide for the Team

1. Determine team size and coverage hours. For 4 people, weekly rotation is optimal.
2. Create a schedule in PagerDuty/OpsGenie: specify participants, rotation type (weekly), handoff time (e.g., Monday 10:00).
3. Configure escalation policy: primary receives alert immediately, if no response within 15 minutes—secondary, another 15 minutes—manager.
4. Configure notification channels: Slack, call, SMS.
5. Run a pilot.

Preparing Runbooks and Handoff Procedure

Runbooks for typical incidents speed up response. Example: on 503—restart Nginx. Handoff note contains: list of open incidents, unstable components, planned changes, 'hot' spots. Template in Slack or Confluence.

Training and Pilot Run

Conduct team training on runbooks and handoff. A one-week pilot will reveal bottlenecks.

Incident Management: How to Combat Alert Fatigue

What Is Alert Fatigue and How to Fight It?

On-call only works if alerts are meaningful. If during an on-call week there are 50 alerts, 45 of which are noise, the team stops responding within a month. Alert fatigue is the on-call engineer's worst enemy. Tools: alert grouping, smart notifications, weekly reviews, SLO-based alerting on burn rate.

On-Call System Health Metrics

Compensation and Retention of On-Call Engineers

On-call is extra load that must be compensated: monetary bonus for on-call week, compensatory day off after a heavy week, payment for each night call. A team without compensation will either sabotage duty or leave.

Timelines and What's Included in a Turnkey Setup

• Designing the rotation scheme considering team size and schedule
• Setting up PagerDuty/OpsGenie: schedules, escalation policies, layers
• Integration with monitoring (Prometheus, Datadog, Sentry)
• Configuring notification channels: Slack, call, SMS
• Documentation: runbook, handoff template, instructions for on-call engineers
• Team training and pilot run

Source: PagerDuty documentation

Setup Timelines

• PagerDuty/OpsGenie + schedules + escalation policy — 1-2 days
• Integration with Prometheus/Datadog alerts — 1-2 days
• Configuration of notification channels (Slack, call, SMS) — 1 day
• Process documentation + team training — 1-2 days

Get a consultation on implementing on-call rotation for your team. Contact us—we'll assess your project and propose a solution within 2 days. Order a turnkey setup, and we guarantee MTTA reduction to 5 minutes.

Website Technical Support: Updates, Monitoring, SLA

A website on Laravel 8 with PHP 7.4. PHP 7.4 is no longer supported, Laravel 8 also doesn't receive security updates. The hosting provider warned about the mandatory PHP update to 8.1 — after the update, two plugins and one library broke, the site went down. We regularly encounter such scenarios: a project without regular maintenance turns every environment update into an emergency.

This case is not an exception, but a rule. Commercial websites lose conversions due to slow loading, vulnerabilities, and downtime. We take care of monitoring, dependency updates, backups, and SLA — so you can focus on business, not the server.

Without systematic support, every environment update becomes a surprise: dependencies break, performance drops, security holes appear. Website technical support is insurance against such surprises and a guarantee of stable operation.

What Actually Goes into Website Technical Support?

Support is not "answering a call when something breaks." It is systematic prevention of breakdowns.

Dependency updates. Composer packages, npm packages, CMS or framework. composer audit and npm audit show known vulnerabilities. Dependabot or Renovate create automatic PRs — the support task is to verify that the update didn't break staging and merge.

Updates types: patch (1.2.3 → 1.2.4, only bugfix, safe), minor (1.2.0 → 1.3.0, new features with backward compatibility, usually safe), major (1.x → 2.x, breaking changes, require testing). Ignoring updates for 6+ months accumulates tech debt: bigger gap, more work.

WordPress is a separate story. The platform's popularity makes it a prime attack target. Outdated plugins are the #1 attack vector. Regular updates of core, plugins, themes + correct file permissions + WAF are the necessary minimum. Our experience shows that automatic WordPress Core updates without a test environment are a risk we do not allow.

How Does Monitoring Prevent Downtime?

Uptime monitoring. Basic HTTP check every minute. Better Uptime, Upptime (self-hosted), Checkly, New Relic Synthetics. Alert to Telegram or Slack on downtime — and notification upon recovery. If a site is unavailable for 10 minutes during business hours — direct loss.

Performance. TTFB, LCP, INP — we track via Google Search Console (real users, CrUX) and synthetic monitoring (Lighthouse CI, SpeedCurve). Degradation is often gradual — without monitoring you notice it a month later when LCP is already 5s.

Application errors. Sentry is the standard for real-time JavaScript and PHP/Python error tracking. Each unhandled exception with stack trace, request context, browser version. Especially important for errors users don't report — they just leave.

Database. Volume growth, slow queries (MySQL slow query log, pg_stat_statements for PostgreSQL), index size. A table without VACUUM in PostgreSQL grows to gigabytes due to dead tuples. Routine database maintenance is part of support.

Disk space and logs. Is logrotate configured? /var/log/nginx growing without limits and filling the disk — classic. Automatic rotation + alert at disk > 80%.

Why Are Backups Without Verification an Illusion?

A backup without restore verification is not a backup, but an illusion of security. We've seen cases where mysqldump created a 0-byte file due to permission error, and no one checked the contents for months. We guarantee all copies are restorable.

Backup scheme:

• Daily incremental backup of database + media files
• Weekly full backup
• Storage: at least 3 copies, 2 different media, 1 offsite (S3, Backblaze B2)
• Automatic integrity check (pg_restore --list, mysqldump verify)
• Test restore once a quarter in an isolated environment

Retention policy: 7 daily, 4 weekly, 3 monthly. S3 Lifecycle rules automate deletion.

SLA: What Does It Mean in Practice?

SLA (Service-Level Agreement) Wikipedia — specific commitments for response and resolution times:

SLA makes sense only with monitoring — otherwise you learn about problems from users, not systems. A broken button in a form can silently kill conversions for weeks.

Content Update Process

A developer should not be in the chain for editing text on a page. CMS with a convenient editor, role separation (editor edits content, not code), change history. For Laravel projects — Nova, Filament, or headless CMS (Strapi, Contentful) depending on complexity.

Preview before publishing, staged rollout for important changes. If editors work directly on prod — that's a risk.

Typical Situations We Resolve

Website hack: attack vector analysis, cleanup, security hardening (WAF, fail2ban, file permission restrictions). Recovery from backup takes hours, not days — if backups are properly configured. Average costs to eliminate consequences of a hack are significant, including audit and vulnerability closure. Regular support is much cheaper and prevents such incidents.

Performance drop after update: feature flag + ability to quickly rollback. Canary deployment — update 5% of traffic, check metrics, then 100%.

What's Included in the Support Package (Deliverables)

Upon signing the contract, you receive:

• Documentation: infrastructure diagram, access, recovery procedures
• Monitoring: uptime, performance, errors, logs — set up from day one
• Backup: daily/weekly copies with verification
• Dependency updates: monthly audit and update with testing
• SLA response: per priorities from the table above
• Reports: weekly dashboards, monthly review, quarterly tech plan
• Content editing support: editor training, permission setup

Contact us to choose a suitable plan and get an initial audit of your project.

How We Work: Stages

1. Onboarding (3–5 days): audit of current state, setup of monitoring and backups, infrastructure documentation.
2. Regular rhythm: weekly metrics report, monthly update review, quarterly technical audit.
3. Response: per SLA, with recording of cause and resolution time.
4. Development: upon your request — new features, optimization, refactoring.

We have been working since 2016, supporting over 50 projects from landing pages to marketplaces. Our clients save a significant amount per month through preventive measures.

Timelines and Cost

Setting up monitoring and backups: 3–5 days. Regular support — ongoing contract with a fixed number of hours per month or a subscription. Cost is calculated individually after audit. Get a consultation — we will evaluate your project in 1–2 days.

Comparison: Monitoring with Automatic Alerting vs Manual Checking

Automatic monitoring with Better Uptime responds to failures 10 times faster than manual checking.