PagerDuty Integration and Incident Management Setup

Your team is drowning in a sea of alerts: 200+ notifications per hour, half are duplicates, the rest are warnings requiring no action. Duty engineers burn out, real incidents get lost, and MTTR grows. We configure PagerDuty so the problem is solved in 2–4 days. Incident management stops being a headache.

One of our projects: Prometheus generated 200+ alerts per hour. After configuring PagerDuty with Event Intelligence, the number of incidents dropped to 5–7 per day. MTTR fell from 45 to 8 minutes — 5 times faster. The average cost of a minute of downtime for e-commerce is high, so the savings were substantial. Over 5+ years, we have completed more than 50 such integrations for teams of various sizes. We guarantee stable system operation after deployment.

PagerDuty Architecture: Key Components

Services — logical units (backend API, payment service, database). Each service has its own escalation policy and on-call schedule.

Integrations — event sources: Prometheus/Alertmanager, Datadog, CloudWatch, Grafana, Uptime Robot, custom webhooks. Each integration generates a unique endpoint key.

Escalation Policies — rules: who gets the alert, after how many minutes escalation, where to escalate.

Schedules — on-call schedules with rotations.

How to Connect Prometheus Alertmanager? (Detailed Case Study from Our Practice)

This is the most common scenario. We proceed step by step:

1. Create a service in PagerDuty and add a Prometheus integration. Get the routing_key.
2. In the Alertmanager config, set up a receiver with this key. Be sure to specify description and severity.
3. Configure grouping by alertname and cluster so related alerts don't spawn multiple incidents.
4. Verify that a test alert arrives in the correct service and escalates according to the policy.

Example receiver configuration:

# alertmanager.yml
route:
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'pagerduty-critical'
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
    - match:
        severity: warning
      receiver: 'slack-warnings'

receivers:
  - name: 'pagerduty-critical'
    pagerduty_configs:
      - routing_key: '<PAGERDUTY_INTEGRATION_KEY>'
        description: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'
        severity: '{{ .CommonLabels.severity }}'
        details:
          firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'

In one project, we configured grouping by cluster and alertname. This reduced the number of created incidents by 70%.

Comparison of Integration Methods

How PagerDuty Helps Tame Alert Noise?

PagerDuty Event Intelligence (available in paid plans) automatically suppresses noise. It includes three mechanisms:

• Alert Grouping: related alerts are merged into one incident. When a DB fails, you won't get 50 alerts from all services — just one.
• Intelligent Alert Grouping: ML model based on historical patterns groups similar alerts.
• Suppression Rules: temporary suppression of alerts during planned maintenance.

According to PagerDuty documentation, noise reduction cuts notifications by up to 90%. In one project, we achieved an 85% reduction.

Advantages of PagerDuty over Email Notifications

Email cannot group alerts, has no escalations, and does not provide MTTR statistics. PagerDuty delivers critical notifications 5 times faster: push notifications arrive in seconds, while email can be delayed by minutes. Moreover, PagerDuty automatically keeps an incident timeline, which aids postmortems.

Table: Key Metrics Before and After PagerDuty Deployment

Automation with Webhooks and Runbooks

PagerDuty Webhooks send events when an incident is created, updated, or resolved. Example handler in Python:

@app.route('/pd-webhook', methods=['POST'])
def pagerduty_webhook():
    data = request.json
    event_type = data['event']['event_type']
    incident = data['event']['data']
    
    if event_type == 'incident.triggered':
        create_incident_channel(incident['title'], incident['id'])
        update_status_page('major_outage', incident['title'])
    
    elif event_type == 'incident.resolved':
        archive_incident_channel(incident['id'])
        update_status_page('operational', '')
    
    return '', 200

Runbook Automation (formerly Rundeck) allows automatic actions on alert: restart service, clean disk, scale. If a script fixes the problem, the incident closes automatically without waking the on-call engineer.

What Does Our Work Include?

• Audit of current monitoring and incident management processes.
• Designing service structure, escalation policies, and schedules.
• Configuring integrations with Prometheus, Datadog, CloudWatch, Grafana, and others.
• Setting up webhooks and automation with Jira/Slack.
• Testing and team training.
• Handover of documentation and access rights.
• Stable operation guarantee and post-deployment support.
• Our engineers have PagerDuty certifications and experience with over 50 integrations.

Get a consultation to optimize your incident management. Contact us for an audit of your system — we will evaluate your infrastructure and suggest an optimal configuration in 1–2 days. Order a turnkey PagerDuty integration and get a 3–5x MTTR reduction.

Website Technical Support: Updates, Monitoring, SLA

A website on Laravel 8 with PHP 7.4. PHP 7.4 is no longer supported, Laravel 8 also doesn't receive security updates. The hosting provider warned about the mandatory PHP update to 8.1 — after the update, two plugins and one library broke, the site went down. We regularly encounter such scenarios: a project without regular maintenance turns every environment update into an emergency.

This case is not an exception, but a rule. Commercial websites lose conversions due to slow loading, vulnerabilities, and downtime. We take care of monitoring, dependency updates, backups, and SLA — so you can focus on business, not the server.

Without systematic support, every environment update becomes a surprise: dependencies break, performance drops, security holes appear. Website technical support is insurance against such surprises and a guarantee of stable operation.

What Actually Goes into Website Technical Support?

Support is not "answering a call when something breaks." It is systematic prevention of breakdowns.

Dependency updates. Composer packages, npm packages, CMS or framework. composer audit and npm audit show known vulnerabilities. Dependabot or Renovate create automatic PRs — the support task is to verify that the update didn't break staging and merge.

Updates types: patch (1.2.3 → 1.2.4, only bugfix, safe), minor (1.2.0 → 1.3.0, new features with backward compatibility, usually safe), major (1.x → 2.x, breaking changes, require testing). Ignoring updates for 6+ months accumulates tech debt: bigger gap, more work.

WordPress is a separate story. The platform's popularity makes it a prime attack target. Outdated plugins are the #1 attack vector. Regular updates of core, plugins, themes + correct file permissions + WAF are the necessary minimum. Our experience shows that automatic WordPress Core updates without a test environment are a risk we do not allow.

How Does Monitoring Prevent Downtime?

Uptime monitoring. Basic HTTP check every minute. Better Uptime, Upptime (self-hosted), Checkly, New Relic Synthetics. Alert to Telegram or Slack on downtime — and notification upon recovery. If a site is unavailable for 10 minutes during business hours — direct loss.

Performance. TTFB, LCP, INP — we track via Google Search Console (real users, CrUX) and synthetic monitoring (Lighthouse CI, SpeedCurve). Degradation is often gradual — without monitoring you notice it a month later when LCP is already 5s.

Application errors. Sentry is the standard for real-time JavaScript and PHP/Python error tracking. Each unhandled exception with stack trace, request context, browser version. Especially important for errors users don't report — they just leave.

Database. Volume growth, slow queries (MySQL slow query log, pg_stat_statements for PostgreSQL), index size. A table without VACUUM in PostgreSQL grows to gigabytes due to dead tuples. Routine database maintenance is part of support.

Disk space and logs. Is logrotate configured? /var/log/nginx growing without limits and filling the disk — classic. Automatic rotation + alert at disk > 80%.

Why Are Backups Without Verification an Illusion?

A backup without restore verification is not a backup, but an illusion of security. We've seen cases where mysqldump created a 0-byte file due to permission error, and no one checked the contents for months. We guarantee all copies are restorable.

Backup scheme:

• Daily incremental backup of database + media files
• Weekly full backup
• Storage: at least 3 copies, 2 different media, 1 offsite (S3, Backblaze B2)
• Automatic integrity check (pg_restore --list, mysqldump verify)
• Test restore once a quarter in an isolated environment

Retention policy: 7 daily, 4 weekly, 3 monthly. S3 Lifecycle rules automate deletion.

SLA: What Does It Mean in Practice?

SLA (Service-Level Agreement) Wikipedia — specific commitments for response and resolution times:

SLA makes sense only with monitoring — otherwise you learn about problems from users, not systems. A broken button in a form can silently kill conversions for weeks.

Content Update Process

A developer should not be in the chain for editing text on a page. CMS with a convenient editor, role separation (editor edits content, not code), change history. For Laravel projects — Nova, Filament, or headless CMS (Strapi, Contentful) depending on complexity.

Preview before publishing, staged rollout for important changes. If editors work directly on prod — that's a risk.

Typical Situations We Resolve

Website hack: attack vector analysis, cleanup, security hardening (WAF, fail2ban, file permission restrictions). Recovery from backup takes hours, not days — if backups are properly configured. Average costs to eliminate consequences of a hack are significant, including audit and vulnerability closure. Regular support is much cheaper and prevents such incidents.

Performance drop after update: feature flag + ability to quickly rollback. Canary deployment — update 5% of traffic, check metrics, then 100%.

What's Included in the Support Package (Deliverables)

Upon signing the contract, you receive:

• Documentation: infrastructure diagram, access, recovery procedures
• Monitoring: uptime, performance, errors, logs — set up from day one
• Backup: daily/weekly copies with verification
• Dependency updates: monthly audit and update with testing
• SLA response: per priorities from the table above
• Reports: weekly dashboards, monthly review, quarterly tech plan
• Content editing support: editor training, permission setup

Contact us to choose a suitable plan and get an initial audit of your project.

How We Work: Stages

1. Onboarding (3–5 days): audit of current state, setup of monitoring and backups, infrastructure documentation.
2. Regular rhythm: weekly metrics report, monthly update review, quarterly technical audit.
3. Response: per SLA, with recording of cause and resolution time.
4. Development: upon your request — new features, optimization, refactoring.

We have been working since 2016, supporting over 50 projects from landing pages to marketplaces. Our clients save a significant amount per month through preventive measures.

Timelines and Cost

Setting up monitoring and backups: 3–5 days. Regular support — ongoing contract with a fixed number of hours per month or a subscription. Cost is calculated individually after audit. Get a consultation — we will evaluate your project in 1–2 days.

Comparison: Monitoring with Automatic Alerting vs Manual Checking

Automatic monitoring with Better Uptime responds to failures 10 times faster than manual checking.