Implementing a Blameless Post-Mortem Process for Incidents

Scenario: Friday evening, payment service goes down, error rate > 5%, 12,000 users cannot complete payment. The team fights the fire for 47 minutes. Root cause: connection pool exhausted because the number of workers increased during deployment, but the pgBouncer config was not updated. A month later—same symptom. This situation is familiar to many SRE engineers. Without a working post-mortem process, incidents recur in 60% of cases. We help implement a blameless post-mortem culture to prevent such errors from repeating.

Key principle: blameless post-mortem culture is not about finding someone to blame, but about improving the system. Our proven methodology, built on over 5 years of experience and 50+ implementations, guarantees results. Our experience: without post-mortems, incidents repeat 60% of the time; with post-mortems, less than 20%. Investigation time is reduced by 30–50% (1.5x–2x faster). SEV1 incidents drop by 60% within six months. Estimated savings: up to $60,000 per year for a team of 10.

Why Blameless Post-Mortem Matters?

Blameless does not mean avoiding accountability. If an engineer made a mistake, the cause is a system that allowed that mistake to happen without safeguards. The correct approach is to ask why the system permitted the error, not who pushed the wrong button. A blame culture leads to hiding incidents and unwillingness to admit mistakes—which is worse than the incident itself. We offer a ready-made template and process that integrates into your incident management system.

When and How to Conduct a Post-Mortem

Identifying Incidents for Review

• SEV1 incidents: always, within 48 hours
• SEV2 incidents: always, within 72 hours
• SEV3: at team's discretion, if it reveals a systemic issue
• Recurring SEV4: worth conducting if the same symptom appears a third time

Post-Mortem Document Structure

Each document includes a timeline, root cause, what went wrong, what went well, and corrective actions. Example timeline:

Meeting Flow

Participants: all incident responders + technical lead + optionally product owner. Duration: 60-90 minutes. Stages: timeline review (10 min), root cause analysis using the 5 Whys technique (20-30 min), improvement discussion (15 min), what went well (5 min), action items with owners and deadlines (15 min).

5 Whys Technique

Example: Connection pool exhausted → Why? Number of connections exceeded max_client_conn → Why? Number of workers increased during deployment → Why? No process to check DB config when scaling → Why? Deployment checklist does not cover config dependencies. Root cause: absence of a process to check configuration dependencies during deployment.

Cause Categories

Documents are tagged with severity, service, cause-category. Main categories: Configuration, Deployment, Dependency failure, Capacity, Human error. A quarterly review helps direct reliability investments.

What Results Does Implementation Bring?

Comparison without and with post-mortem:

Reducing SEV1 frequency by 60% saves up to $60,000 per year for a team of 10 (average SEV1 incident costs $10,000). Cutting investigation time by 30–50% saves 10+ person-hours per week.

How to Ensure Action Items Are Completed?

A post-mortem is useless if action items are not executed. Required conditions:

• Specific owner (not "team" but a name)
• Clear deadline
• Ticket created in Jira/Linear immediately during the meeting
• Review completion at the next post-mortem meeting

For analytics, use a dashboard with cause categories to identify systemic issues. Integration with PagerDuty helps automate metric collection.

Post-Mortem Implementation Process

1. Audit current incident management process and identify growth areas
2. Develop a post-mortem template tailored to your infrastructure and stack
3. Train the team: workshop on blameless culture and the 5 Whys technique
4. Pilot a post-mortem on a real incident with mentoring
5. Integrate with ticketing system (Jira/Linear) and configure automatic metric collection
6. Quarterly review of results and process adjustment

What's Included in the Work

• Ready-to-use post-mortem document template in Confluence/Notion
• Team training (up to 2 hours)
• Integration with Jira/Linear: automatic creation of action items
• Dashboard setup for cause analytics
• Support for one month after implementation

Estimated Timelines

• Basic implementation: 3 to 5 days
• Full cycle with training and setup: 5 to 10 days

Pricing is calculated individually. To get a consultation and accurate estimate, contact us. Order a pilot project—we will analyze one incident and show the result.

Website Technical Support: Updates, Monitoring, SLA

A website on Laravel 8 with PHP 7.4. PHP 7.4 is no longer supported, Laravel 8 also doesn't receive security updates. The hosting provider warned about the mandatory PHP update to 8.1 — after the update, two plugins and one library broke, the site went down. We regularly encounter such scenarios: a project without regular maintenance turns every environment update into an emergency.

This case is not an exception, but a rule. Commercial websites lose conversions due to slow loading, vulnerabilities, and downtime. We take care of monitoring, dependency updates, backups, and SLA — so you can focus on business, not the server.

Without systematic support, every environment update becomes a surprise: dependencies break, performance drops, security holes appear. Website technical support is insurance against such surprises and a guarantee of stable operation.

What Actually Goes into Website Technical Support?

Support is not "answering a call when something breaks." It is systematic prevention of breakdowns.

Dependency updates. Composer packages, npm packages, CMS or framework. composer audit and npm audit show known vulnerabilities. Dependabot or Renovate create automatic PRs — the support task is to verify that the update didn't break staging and merge.

Updates types: patch (1.2.3 → 1.2.4, only bugfix, safe), minor (1.2.0 → 1.3.0, new features with backward compatibility, usually safe), major (1.x → 2.x, breaking changes, require testing). Ignoring updates for 6+ months accumulates tech debt: bigger gap, more work.

WordPress is a separate story. The platform's popularity makes it a prime attack target. Outdated plugins are the #1 attack vector. Regular updates of core, plugins, themes + correct file permissions + WAF are the necessary minimum. Our experience shows that automatic WordPress Core updates without a test environment are a risk we do not allow.

How Does Monitoring Prevent Downtime?

Uptime monitoring. Basic HTTP check every minute. Better Uptime, Upptime (self-hosted), Checkly, New Relic Synthetics. Alert to Telegram or Slack on downtime — and notification upon recovery. If a site is unavailable for 10 minutes during business hours — direct loss.

Performance. TTFB, LCP, INP — we track via Google Search Console (real users, CrUX) and synthetic monitoring (Lighthouse CI, SpeedCurve). Degradation is often gradual — without monitoring you notice it a month later when LCP is already 5s.

Application errors. Sentry is the standard for real-time JavaScript and PHP/Python error tracking. Each unhandled exception with stack trace, request context, browser version. Especially important for errors users don't report — they just leave.

Database. Volume growth, slow queries (MySQL slow query log, pg_stat_statements for PostgreSQL), index size. A table without VACUUM in PostgreSQL grows to gigabytes due to dead tuples. Routine database maintenance is part of support.

Disk space and logs. Is logrotate configured? /var/log/nginx growing without limits and filling the disk — classic. Automatic rotation + alert at disk > 80%.

Why Are Backups Without Verification an Illusion?

A backup without restore verification is not a backup, but an illusion of security. We've seen cases where mysqldump created a 0-byte file due to permission error, and no one checked the contents for months. We guarantee all copies are restorable.

Backup scheme:

• Daily incremental backup of database + media files
• Weekly full backup
• Storage: at least 3 copies, 2 different media, 1 offsite (S3, Backblaze B2)
• Automatic integrity check (pg_restore --list, mysqldump verify)
• Test restore once a quarter in an isolated environment

Retention policy: 7 daily, 4 weekly, 3 monthly. S3 Lifecycle rules automate deletion.

SLA: What Does It Mean in Practice?

SLA (Service-Level Agreement) Wikipedia — specific commitments for response and resolution times:

SLA makes sense only with monitoring — otherwise you learn about problems from users, not systems. A broken button in a form can silently kill conversions for weeks.

Content Update Process

A developer should not be in the chain for editing text on a page. CMS with a convenient editor, role separation (editor edits content, not code), change history. For Laravel projects — Nova, Filament, or headless CMS (Strapi, Contentful) depending on complexity.

Preview before publishing, staged rollout for important changes. If editors work directly on prod — that's a risk.

Typical Situations We Resolve

Website hack: attack vector analysis, cleanup, security hardening (WAF, fail2ban, file permission restrictions). Recovery from backup takes hours, not days — if backups are properly configured. Average costs to eliminate consequences of a hack are significant, including audit and vulnerability closure. Regular support is much cheaper and prevents such incidents.

Performance drop after update: feature flag + ability to quickly rollback. Canary deployment — update 5% of traffic, check metrics, then 100%.

What's Included in the Support Package (Deliverables)

Upon signing the contract, you receive:

• Documentation: infrastructure diagram, access, recovery procedures
• Monitoring: uptime, performance, errors, logs — set up from day one
• Backup: daily/weekly copies with verification
• Dependency updates: monthly audit and update with testing
• SLA response: per priorities from the table above
• Reports: weekly dashboards, monthly review, quarterly tech plan
• Content editing support: editor training, permission setup

Contact us to choose a suitable plan and get an initial audit of your project.

How We Work: Stages

1. Onboarding (3–5 days): audit of current state, setup of monitoring and backups, infrastructure documentation.
2. Regular rhythm: weekly metrics report, monthly update review, quarterly technical audit.
3. Response: per SLA, with recording of cause and resolution time.
4. Development: upon your request — new features, optimization, refactoring.

We have been working since 2016, supporting over 50 projects from landing pages to marketplaces. Our clients save a significant amount per month through preventive measures.

Timelines and Cost

Setting up monitoring and backups: 3–5 days. Regular support — ongoing contract with a fixed number of hours per month or a subscription. Cost is calculated individually after audit. Get a consultation — we will evaluate your project in 1–2 days.

Comparison: Monitoring with Automatic Alerting vs Manual Checking

Automatic monitoring with Better Uptime responds to failures 10 times faster than manual checking.