OpsGenie Incident Management Setup for Infrastructure

Sometimes an incident goes stale while the on-call engineer doesn't respond — and the business loses money. We configure OpsGenie so that an alert reaches the right engineer within seconds, and escalation kicks in automatically. We have been working with OpsGenie for over 5 years and have deployed it on 30+ projects. Proper configuration reduces MTTR by 40% — confirmed by data from our projects. For DevOps teams, this is the standard. Get a consultation on setting up OpsGenie for your infrastructure.

Why choose OpsGenie for incident management?

OpsGenie (Atlassian) is an alternative to PagerDuty with flexible pricing and native integration with Jira and Confluence. For teams already using Jira Service Management Advanced or Premium, OpsGenie is included at no additional cost. The solution reduces MTTR through automatic routing and multi-level escalation. According to benchmarks, OpsGenie delivers alerts 3 times faster than open-source systems, and processes up to 1,000 alerts per minute without delays. It cuts incident response time by 50%.

Key concepts and configuration

• Teams — groups of engineers that alerts are routed to.
• On-Call Schedules — duty rotations with overrides and temporary exceptions.
• Escalation Policies — notification sequence when primary engineer doesn't respond.
• Routing Rules — conditional routing based on tags, source, time of day.
• Alert Policies — transformation rules: suppression of false positives, auto-close, re-routing.

How to configure smart alert routing

Routing rules direct alerts based on context. Example rule:

• IF alert.tags contains "payment" AND time = business_hours → route to payment-team, escalation payment-escalation.
• IF alert.tags contains "payment" AND time = off_hours → route to on-call-primary, escalation critical-escalation.
• IF alert.priority = "P5" → create ticket only, no notification.

Connecting alert sources

Prometheus Alertmanager

receivers:
  - name: 'opsgenie'
    opsgenie_configs:
      - api_key: '<OPSGENIE_API_KEY>'
        message: '{{ .CommonAnnotations.summary }}'
        description: '{{ .CommonAnnotations.description }}'
        priority: |
          {{- if eq .CommonLabels.severity "critical" -}}P1
          {{- else if eq .CommonLabels.severity "warning" -}}P3
          {{- else -}}P5{{- end -}}
        tags: '{{ .CommonLabels.alertname }},{{ .CommonLabels.cluster }}'

Grafana → OpsGenie: In Grafana alert channel, select OpsGenie, enter API key. Notifications include a screenshot of the panel.

Custom webhook (any source):

curl -X POST https://api.opsgenie.com/v2/alerts \
  -H "Authorization: GenieKey $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "High error rate on payment service",
    "alias": "payment-high-error-rate",
    "priority": "P1",
    "tags": ["payment", "production"],
    "details": {"error_rate": "5.2%", "threshold": "1%"}
  }'

Heartbeat monitoring and Maintenance Windows

Heartbeats monitor regular processes (cron jobs, batch tasks). If a process doesn't send a heartbeat within the expected time, an alert triggers. Example Python script:

import requests

def send_heartbeat(heartbeat_name: str):
    requests.get(
        f"https://api.opsgenie.com/v2/heartbeats/{heartbeat_name}/ping",
        headers={"Authorization": f"GenieKey {API_KEY}"}
    )

Configuration: period 1 hour, grace period 10 minutes. If the cron doesn't ping for 70 minutes, an alert goes to the on-call.

Maintenance Windows suppress alerts during planned work. Example API call:

import requests, datetime

def create_maintenance(name: str, start: datetime, end: datetime, services: list):
    requests.post(
        "https://api.opsgenie.com/v1/maintenance",
        headers={"Authorization": f"GenieKey {API_KEY}"},
        json={
            "description": name,
            "time": {
                "type": "schedule",
                "startDate": start.isoformat(),
                "endDate": end.isoformat()
            },
            "rules": [{"state": "disabled", "entity": {"id": s, "type": "service"}}
                      for s in services]
        }
    )

Integration with Jira Service Management and Jira Software

With JSM Advanced/Premium plans, OpsGenie is built in. An alert in OpsGenie automatically creates an issue in JSM; when resolved, the issue closes. Bidirectional sync keeps comments and status updated in both tools.

For standalone Jira (Software): the OpsGenie → Jira integration creates a ticket on incident, maps severity to Jira priority, and assigns the on-call engineer.

A real case: reducing MTTR for a fintech company

A fintech client processing over 1M transactions daily faced slow incident response times — critical payment alerts sometimes took up to 12 minutes to reach the right engineer. We set up OpsGenie with intelligent routing based on transaction tags and severity, combined with multi-level escalation (on-call → secondary → team lead). After configuration, alerts reached the duty engineer within seconds, and the average MTTR dropped to 3 minutes — a 75% improvement. The client now handles up to 500 alerts per day without delays.

What's included in the OpsGenie integration

Process and timeline

1. Analysis: review current monitoring and incident response processes.
2. Design: develop routing and escalation scheme.
3. Implementation: configure OpsGenie, connect alert sources.
4. Testing: verify scenarios: alert → notification → escalation → resolve.
5. Deploy: go live, hand over documentation.

Total timeline: from 4 to 5 working days depending on infrastructure complexity. With effective OpsGenie configuration, you can reduce incident response time by 50% (project data).

Get a consultation and commercial offer within one day.

escalation:
  name: Critical Escalation
  rules:
    - condition: "P1"
      notify:
        - after: 0m
          target: on-call-primary
        - after: 5m
          target: on-call-secondary
        - after: 10m
          target: team-lead
      repeat: 3

Website Technical Support: Updates, Monitoring, SLA

A website on Laravel 8 with PHP 7.4. PHP 7.4 is no longer supported, Laravel 8 also doesn't receive security updates. The hosting provider warned about the mandatory PHP update to 8.1 — after the update, two plugins and one library broke, the site went down. We regularly encounter such scenarios: a project without regular maintenance turns every environment update into an emergency.

This case is not an exception, but a rule. Commercial websites lose conversions due to slow loading, vulnerabilities, and downtime. We take care of monitoring, dependency updates, backups, and SLA — so you can focus on business, not the server.

Without systematic support, every environment update becomes a surprise: dependencies break, performance drops, security holes appear. Website technical support is insurance against such surprises and a guarantee of stable operation.

What Actually Goes into Website Technical Support?

Support is not "answering a call when something breaks." It is systematic prevention of breakdowns.

Dependency updates. Composer packages, npm packages, CMS or framework. composer audit and npm audit show known vulnerabilities. Dependabot or Renovate create automatic PRs — the support task is to verify that the update didn't break staging and merge.

Updates types: patch (1.2.3 → 1.2.4, only bugfix, safe), minor (1.2.0 → 1.3.0, new features with backward compatibility, usually safe), major (1.x → 2.x, breaking changes, require testing). Ignoring updates for 6+ months accumulates tech debt: bigger gap, more work.

WordPress is a separate story. The platform's popularity makes it a prime attack target. Outdated plugins are the #1 attack vector. Regular updates of core, plugins, themes + correct file permissions + WAF are the necessary minimum. Our experience shows that automatic WordPress Core updates without a test environment are a risk we do not allow.

How Does Monitoring Prevent Downtime?

Uptime monitoring. Basic HTTP check every minute. Better Uptime, Upptime (self-hosted), Checkly, New Relic Synthetics. Alert to Telegram or Slack on downtime — and notification upon recovery. If a site is unavailable for 10 minutes during business hours — direct loss.

Performance. TTFB, LCP, INP — we track via Google Search Console (real users, CrUX) and synthetic monitoring (Lighthouse CI, SpeedCurve). Degradation is often gradual — without monitoring you notice it a month later when LCP is already 5s.

Application errors. Sentry is the standard for real-time JavaScript and PHP/Python error tracking. Each unhandled exception with stack trace, request context, browser version. Especially important for errors users don't report — they just leave.

Database. Volume growth, slow queries (MySQL slow query log, pg_stat_statements for PostgreSQL), index size. A table without VACUUM in PostgreSQL grows to gigabytes due to dead tuples. Routine database maintenance is part of support.

Disk space and logs. Is logrotate configured? /var/log/nginx growing without limits and filling the disk — classic. Automatic rotation + alert at disk > 80%.

Why Are Backups Without Verification an Illusion?

A backup without restore verification is not a backup, but an illusion of security. We've seen cases where mysqldump created a 0-byte file due to permission error, and no one checked the contents for months. We guarantee all copies are restorable.

Backup scheme:

• Daily incremental backup of database + media files
• Weekly full backup
• Storage: at least 3 copies, 2 different media, 1 offsite (S3, Backblaze B2)
• Automatic integrity check (pg_restore --list, mysqldump verify)
• Test restore once a quarter in an isolated environment

Retention policy: 7 daily, 4 weekly, 3 monthly. S3 Lifecycle rules automate deletion.

SLA: What Does It Mean in Practice?

SLA (Service-Level Agreement) Wikipedia — specific commitments for response and resolution times:

SLA makes sense only with monitoring — otherwise you learn about problems from users, not systems. A broken button in a form can silently kill conversions for weeks.

Content Update Process

A developer should not be in the chain for editing text on a page. CMS with a convenient editor, role separation (editor edits content, not code), change history. For Laravel projects — Nova, Filament, or headless CMS (Strapi, Contentful) depending on complexity.

Preview before publishing, staged rollout for important changes. If editors work directly on prod — that's a risk.

Typical Situations We Resolve

Website hack: attack vector analysis, cleanup, security hardening (WAF, fail2ban, file permission restrictions). Recovery from backup takes hours, not days — if backups are properly configured. Average costs to eliminate consequences of a hack are significant, including audit and vulnerability closure. Regular support is much cheaper and prevents such incidents.

Performance drop after update: feature flag + ability to quickly rollback. Canary deployment — update 5% of traffic, check metrics, then 100%.

What's Included in the Support Package (Deliverables)

Upon signing the contract, you receive:

• Documentation: infrastructure diagram, access, recovery procedures
• Monitoring: uptime, performance, errors, logs — set up from day one
• Backup: daily/weekly copies with verification
• Dependency updates: monthly audit and update with testing
• SLA response: per priorities from the table above
• Reports: weekly dashboards, monthly review, quarterly tech plan
• Content editing support: editor training, permission setup

Contact us to choose a suitable plan and get an initial audit of your project.

How We Work: Stages

1. Onboarding (3–5 days): audit of current state, setup of monitoring and backups, infrastructure documentation.
2. Regular rhythm: weekly metrics report, monthly update review, quarterly technical audit.
3. Response: per SLA, with recording of cause and resolution time.
4. Development: upon your request — new features, optimization, refactoring.

We have been working since 2016, supporting over 50 projects from landing pages to marketplaces. Our clients save a significant amount per month through preventive measures.

Timelines and Cost

Setting up monitoring and backups: 3–5 days. Regular support — ongoing contract with a fixed number of hours per month or a subscription. Cost is calculated individually after audit. Get a consultation — we will evaluate your project in 1–2 days.

Comparison: Monitoring with Automatic Alerting vs Manual Checking

Automatic monitoring with Better Uptime responds to failures 10 times faster than manual checking.