Website Recovery: Runbook, Backups, and Timelines

Website Recovery After Failure

You discovered that the site is not opening and SSH access is lost? Or the last backup turned out to be corrupt? In 10 years we have restored more than 50 projects: from WordPress hacks to complete disk array failure. The key to fast recovery is not heroism, but a prepared runbook and regularly tested backups. Without this, recovery can take days instead of hours. In this article you will find ready-made diagnostic and recovery scripts, as well as tips for automating backups. Our experience confirms: proper preparation reduces downtime by 5 times.

Main Failure Scenarios and Diagnostics

Diagnostics is the first step. Here is a standard set of commands we run when the site is unavailable:

# Check services and logs
systemctl status nginx php8.2-fpm mysql
journalctl -u nginx -n 100 --no-pager
tail -100 /var/log/php8.2-fpm.log
# Disk and memory diagnostics
df -h
du -sh /var/log/* | sort -rh | head -10
dmesg | grep -i "out of memory"
free -m
# If database is not responding
sudo systemctl restart postgresql
tail -50 /var/log/postgresql/postgresql-14-main.log

Typical causes: full disk (clear logs, temporary Docker files), insufficient memory (add swap), corrupted database tables. After diagnostics, choose a recovery scenario. If no fresh backup is available, don't panic — often a rollback to the last stable dump from cloud storage helps. Make sure the DBMS version matches, otherwise recovery will fail.

Why Is Backup Testing Critical?

30% of backups we check in production turn out to be useless: archive corrupted, dump incomplete, DBMS version mismatch. We test recovery on a staging environment before relying on a backup. Testing takes 1-2 hours but can save days of downtime. Regular tests are the only way to ensure your Disaster Recovery Plan works. According to Wikipedia, testing is a mandatory part of the plan.

We recommend a combination: weekly full + daily incremental. Recovery from an incremental backup takes 2-3 times longer than from a full one, but saves space.

Recovery from Backup: Step-by-Step (Our Case)

Let's break down a real situation: a client updated a plugin, the site went down, and the backup was two weeks old. Here's what we did:

# Identify the last working backup
aws s3 ls s3://my-backups/database/ | tail -5
# Download
aws s3 cp s3://my-backups/database/mysite_<backup_date>.dump.gz /tmp/
# Create a new DB (leave old one untouched — test first)
createdb mysite_restored
gunzip < /tmp/mysite_<backup_date>.dump.gz | pg_restore -d mysite_restored --no-owner
# Check integrity
psql -d mysite_restored -c "SELECT COUNT(*) FROM users;"
psql -d mysite_restored -c "SELECT MAX(created_at) FROM orders;"
# Switch application to restored DB (change DB_NAME in .env)
# If all good — rename database
# ALTER DATABASE mysite RENAME TO mysite_broken;
# ALTER DATABASE mysite_restored RENAME TO mysite;

After recovery, we configured daily incremental backups to Amazon S3 and automated testing via pgBackRest.

How to Automate Backup Testing?

Manual testing is expensive. We use a script that once a week deploys a backup to an isolated staging environment and checks data integrity. If the test fails, the team gets an alert in Slack. Automating testing reduces the risk of discovering a broken backup during an emergency by 80%.

File Recovery and Code Rollback

If only files are damaged (e.g., media), use cloud synchronization:

aws s3 sync s3://my-backups/files/ /var/www/mysite/storage/app/public/ \
  --exact-timestamps
chown -R www-data:www-data /var/www/mysite/storage/
chmod -R 755 /var/www/mysite/storage/

For code rollback, use Git or Docker:

# Via Git
git log --oneline -10
git checkout <commit-hash>
# or
git revert <bad-commit>
# Via Docker
docker pull myregistry/myapp:previous-tag
docker stop myapp
docker run -d --name myapp myregistry/myapp:previous-tag

How to Prepare a Runbook for Fast Recovery?

A runbook is a step-by-step guide for the team. Without it, you waste up to 3 times more time in a stressful situation. We include in the runbook:

• Service check scripts
• Recovery scenarios for each failure type
• Contacts of responsible persons and notification channels
• Post-mortem template

Example runbook table:

What's Included in the Recovery Service

• Audit of current backup scheme — we check what is saved and restored.
• Runbook preparation — detailed instructions for your stack.
• Recovery testing on a staging environment — we prove the process works.
• Backup optimization — incremental, off-site, encrypted.
• Team training — we hold a session to practice failure scenario.
• Post-recovery support — we guarantee stability for 30 days.

Timelines and Cost

Website recovery with a ready runbook — from 1 to 3 days depending on complexity. Full audit and Disaster Recovery Plan setup — from 5 to 10 business days. Cost is calculated individually after the audit.

We are confident in our quality — we guarantee results. Assess your recovery plan now: contact us for a free audit. Get a readiness checklist and recommendations. Order recovery today — our engineers will contact you within an hour.

Our numbers: 10+ years of experience, 50+ restored projects, average recovery time 45 minutes. Recovery from a tested backup is 5 times faster than from an untested one.

Website Technical Support: Updates, Monitoring, SLA

A website on Laravel 8 with PHP 7.4. PHP 7.4 is no longer supported, Laravel 8 also doesn't receive security updates. The hosting provider warned about the mandatory PHP update to 8.1 — after the update, two plugins and one library broke, the site went down. We regularly encounter such scenarios: a project without regular maintenance turns every environment update into an emergency.

This case is not an exception, but a rule. Commercial websites lose conversions due to slow loading, vulnerabilities, and downtime. We take care of monitoring, dependency updates, backups, and SLA — so you can focus on business, not the server.

Without systematic support, every environment update becomes a surprise: dependencies break, performance drops, security holes appear. Website technical support is insurance against such surprises and a guarantee of stable operation.

What Actually Goes into Website Technical Support?

Support is not "answering a call when something breaks." It is systematic prevention of breakdowns.

Dependency updates. Composer packages, npm packages, CMS or framework. composer audit and npm audit show known vulnerabilities. Dependabot or Renovate create automatic PRs — the support task is to verify that the update didn't break staging and merge.

Updates types: patch (1.2.3 → 1.2.4, only bugfix, safe), minor (1.2.0 → 1.3.0, new features with backward compatibility, usually safe), major (1.x → 2.x, breaking changes, require testing). Ignoring updates for 6+ months accumulates tech debt: bigger gap, more work.

WordPress is a separate story. The platform's popularity makes it a prime attack target. Outdated plugins are the #1 attack vector. Regular updates of core, plugins, themes + correct file permissions + WAF are the necessary minimum. Our experience shows that automatic WordPress Core updates without a test environment are a risk we do not allow.

How Does Monitoring Prevent Downtime?

Uptime monitoring. Basic HTTP check every minute. Better Uptime, Upptime (self-hosted), Checkly, New Relic Synthetics. Alert to Telegram or Slack on downtime — and notification upon recovery. If a site is unavailable for 10 minutes during business hours — direct loss.

Performance. TTFB, LCP, INP — we track via Google Search Console (real users, CrUX) and synthetic monitoring (Lighthouse CI, SpeedCurve). Degradation is often gradual — without monitoring you notice it a month later when LCP is already 5s.

Application errors. Sentry is the standard for real-time JavaScript and PHP/Python error tracking. Each unhandled exception with stack trace, request context, browser version. Especially important for errors users don't report — they just leave.

Database. Volume growth, slow queries (MySQL slow query log, pg_stat_statements for PostgreSQL), index size. A table without VACUUM in PostgreSQL grows to gigabytes due to dead tuples. Routine database maintenance is part of support.

Disk space and logs. Is logrotate configured? /var/log/nginx growing without limits and filling the disk — classic. Automatic rotation + alert at disk > 80%.

Why Are Backups Without Verification an Illusion?

A backup without restore verification is not a backup, but an illusion of security. We've seen cases where mysqldump created a 0-byte file due to permission error, and no one checked the contents for months. We guarantee all copies are restorable.

Backup scheme:

• Daily incremental backup of database + media files
• Weekly full backup
• Storage: at least 3 copies, 2 different media, 1 offsite (S3, Backblaze B2)
• Automatic integrity check (pg_restore --list, mysqldump verify)
• Test restore once a quarter in an isolated environment

Retention policy: 7 daily, 4 weekly, 3 monthly. S3 Lifecycle rules automate deletion.

SLA: What Does It Mean in Practice?

SLA (Service-Level Agreement) Wikipedia — specific commitments for response and resolution times:

SLA makes sense only with monitoring — otherwise you learn about problems from users, not systems. A broken button in a form can silently kill conversions for weeks.

Content Update Process

A developer should not be in the chain for editing text on a page. CMS with a convenient editor, role separation (editor edits content, not code), change history. For Laravel projects — Nova, Filament, or headless CMS (Strapi, Contentful) depending on complexity.

Preview before publishing, staged rollout for important changes. If editors work directly on prod — that's a risk.

Typical Situations We Resolve

Website hack: attack vector analysis, cleanup, security hardening (WAF, fail2ban, file permission restrictions). Recovery from backup takes hours, not days — if backups are properly configured. Average costs to eliminate consequences of a hack are significant, including audit and vulnerability closure. Regular support is much cheaper and prevents such incidents.

Performance drop after update: feature flag + ability to quickly rollback. Canary deployment — update 5% of traffic, check metrics, then 100%.

What's Included in the Support Package (Deliverables)

Upon signing the contract, you receive:

• Documentation: infrastructure diagram, access, recovery procedures
• Monitoring: uptime, performance, errors, logs — set up from day one
• Backup: daily/weekly copies with verification
• Dependency updates: monthly audit and update with testing
• SLA response: per priorities from the table above
• Reports: weekly dashboards, monthly review, quarterly tech plan
• Content editing support: editor training, permission setup

Contact us to choose a suitable plan and get an initial audit of your project.

How We Work: Stages

1. Onboarding (3–5 days): audit of current state, setup of monitoring and backups, infrastructure documentation.
2. Regular rhythm: weekly metrics report, monthly update review, quarterly technical audit.
3. Response: per SLA, with recording of cause and resolution time.
4. Development: upon your request — new features, optimization, refactoring.

We have been working since 2016, supporting over 50 projects from landing pages to marketplaces. Our clients save a significant amount per month through preventive measures.

Timelines and Cost

Setting up monitoring and backups: 3–5 days. Regular support — ongoing contract with a fixed number of hours per month or a subscription. Cost is calculated individually after audit. Get a consultation — we will evaluate your project in 1–2 days.

Comparison: Monitoring with Automatic Alerting vs Manual Checking

Automatic monitoring with Better Uptime responds to failures 10 times faster than manual checking.