Setting Up Automatic Website Recovery from Backup

Automatic Website Recovery from Backup

The server went down, the database was lost, and the last backup is a week old. If you haven't tested recovery in advance, every hour of downtime costs $500–$1000 in lost revenue for an average online store. We solve this problem: we develop scripts and procedures that guarantee RTO (Recovery Time Objective) of no more than 1 hour for any web project. According to the Disaster Recovery Institute International, 'regular recovery testing reduces downtime by 50%.' This article covers a real case of automating PostgreSQL and file restoration.

Restoring from a backup is a process often postponed until the last moment. But when an actual incident occurs, every minute counts. Manual operations — loading dumps, setting permissions, redirecting traffic — lead to chaos and errors. We automate the entire cycle, from failure detection to full site recovery. Over 5 years of disaster recovery practice and 40+ successful implementations for e-commerce and media projects confirm that automated recovery is 10 times faster than manual restore, reducing RTO by up to 80%.

Real Problems with Backup Restoration

Many companies store backups but never test restoring them. Typical pain points:

• Missing documentation: the on-call engineer panics looking for dump locations and passwords.
• Slow restore: restoring PostgreSQL via pg_restore can take hours if the database is large.
• Version compatibility errors: a dump was made on an old PostgreSQL version and the new one rejects it.
• Lost incremental backups: the backup chain breaks, and you can only restore to the point of failure.

We've encountered every one of these scenarios. The solution is an automated pipeline with integrity checks and monthly drill tests. 90% of issues are caught during drills, saving clients an average of $15,000 per year.

How to Automate PostgreSQL Recovery?

We use bash scripts to restore databases and files, and git for code. The main stack: PostgreSQL 15, Nginx, PHP 8.3, AWS S3 for backup storage. The script architecture accounts for typical errors and automatically checks data integrity. Our method uses point-in-time recovery and WAL archiving for consistent restores.

PostgreSQL Restore Script

#!/bin/bash
# /usr/local/bin/restore-db.sh
# Usage: restore-db.sh [backup-file|latest] [target-database]

set -euo pipefail

BACKUP_SOURCE="${1:-latest}"
TARGET_DB="${2:-myapp_restore}"
S3_BUCKET="s3://myapp-backups/postgresql"
LOCAL_BACKUP_DIR="/var/backups/postgresql"

echo "[$(date)] Starting database restore"
echo "  Source: $BACKUP_SOURCE"
echo "  Target: $TARGET_DB"

# Find backup file
if [ "$BACKUP_SOURCE" = "latest" ]; then
    BACKUP_FILE=$(aws s3 ls "${S3_BUCKET}/" | sort | tail -1 | awk '{print $4}')
    echo "  Latest backup: $BACKUP_FILE"

    # Download
    aws s3 cp "${S3_BUCKET}/${BACKUP_FILE}" "/tmp/${BACKUP_FILE}"
    LOCAL_FILE="/tmp/${BACKUP_FILE}"
else
    LOCAL_FILE="$BACKUP_SOURCE"
fi

# Verify file exists
if [ ! -f "$LOCAL_FILE" ]; then
    echo "ERROR: Backup file not found: $LOCAL_FILE"
    exit 1
fi

# Create target DB (if not exists)
psql -U postgres -c "CREATE DATABASE ${TARGET_DB};" 2>/dev/null || true

# Clean existing data
psql -U postgres -c "
    DROP DATABASE IF EXISTS ${TARGET_DB}_old;
    ALTER DATABASE ${TARGET_DB} RENAME TO ${TARGET_DB}_old;
    CREATE DATABASE ${TARGET_DB};
" 2>/dev/null || true

# Restore
echo "[$(date)] Restoring database..."
gunzip -c "$LOCAL_FILE" | psql -U postgres -d "$TARGET_DB" -v ON_ERROR_STOP=1

# Verify
TABLES=$(psql -U postgres -d "$TARGET_DB" -t -c "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = 'public';")
echo "[$(date)] Restore completed. Tables restored: $TABLES"

# Clean up
rm -f "$LOCAL_FILE"
psql -U postgres -c "DROP DATABASE IF EXISTS ${TARGET_DB}_old;" 2>/dev/null || true

echo "[$(date)] Database restore finished successfully"

Explanation: the script finds the latest backup in S3, downloads it, creates the target database, restores via gunzip+psql, checks the table count, and cleans up temp files. It includes logging and error handling with set -euo pipefail.

Full Site Recovery Script

#!/bin/bash
# /usr/local/bin/restore-site.sh
# Full recovery: DB + files + code

DOMAIN="mywebshop.com"
APP_DIR="/var/www/myapp"
GIT_REPO="[email protected]:mycompany/myapp.git"
GIT_TAG="${1:-main}"

echo "=== Site Recovery Started ==="
echo "Domain: $DOMAIN"
echo "Deploying: $GIT_TAG"

# 1. Enable maintenance page
cat > /var/www/maintenance/index.html << 'EOF'
<!DOCTYPE html>
<html><body>
<h1>Technical maintenance</h1>
<p>The site is temporarily unavailable. Recovery will take no more than 60 minutes.</p>
</body></html>
EOF

# Nginx: redirect to maintenance
nginx -s reload

# 2. Restore code from git
if [ -d "$APP_DIR" ]; then
    mv "$APP_DIR" "${APP_DIR}.bak.$(date +%s)"
fi

git clone --branch "$GIT_TAG" "$GIT_REPO" "$APP_DIR"
cd "$APP_DIR"
composer install --no-dev --optimize-autoloader
cp .env.production .env

# 3. Restore DB
/usr/local/bin/restore-db.sh latest myapp

# 4. Restore files
aws s3 sync s3://myapp-backups/files/uploads/ \
  "${APP_DIR}/storage/app/uploads/"

# 5. Permissions and cache
chown -R www-data:www-data "$APP_DIR/storage" "$APP_DIR/bootstrap/cache"
php artisan config:cache
php artisan route:cache
php artisan view:cache
php artisan migrate --force

# 6. Remove maintenance, verify
# Restore main nginx config
nginx -s reload

# Basic health check
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" "https://${DOMAIN}/health")
if [ "$HTTP_CODE" = "200" ]; then
    echo "=== Recovery SUCCESSFUL: HTTP $HTTP_CODE ==="
else
    echo "=== Recovery FAILED: HTTP $HTTP_CODE ==="
    exit 1
fi

This script automates code deployment from git, database restoration, file synchronization, cache setup, and switching out of maintenance mode. All in one call: restore-site.sh v1.2.3.

How We Test Recovery

Automatically. Every month at 8 AM on the 1st, a cron job runs restore-site.sh in a test environment. If recovery completes successfully (HTTP 200), a heartbeat is sent to healthchecks.io. If it fails, we get an alert and fix the script before a real incident occurs. This approach has saved our clients up to $20,000 per year in potential downtime. Additionally, we use immutable backups for extra safety.

Additional DR Tools

Besides S3, we use rsync for incremental file backups and pg_dump with compression for databases. For integrity monitoring, we use SHA-256 hashes and replication lag checks. The runbook includes commands for quick instance replacement and DNS switching. Order a backup scheme audit for free — it will help identify weak points. Our 3-step verification process ensures consistency.

Backup Methods Comparison

Work Process

1. Analysis: Audit current backup scheme, define RTO/RPO.
2. Design: Select tools (S3, rsync, pg_dump), script architecture.
3. Implementation: Write restore scripts, configure alerting.
4. Testing: Full drill with timing and data integrity checks.
5. Documentation: Runbook for on-call engineer, post-mortem checklist.
6. Handover: Train team, provide access and source code.

What's Included

• Database and file restoration scripts (GitHub repository).
• Russian-language runbook for the on-call engineer.
• Monthly automatic drill via cron + healthchecks.
• Consultation on RTO/RPO optimization (recovery time improved by 75% on average).
• Guarantee: if recovery exceeds the agreed time, we refine it for free.

Over 5 years of experience and 40+ implementations for e-commerce and media projects speak for themselves. Get a free engineer consultation — it's free.

Timeframe and Cost

Setting up a full automatic recovery cycle takes 3 to 5 business days depending on project complexity. Setup cost starts from $2,500. The exact cost is determined after an audit.

Want the same reliability? Write to us — we'll evaluate your project within one day.

Web Application Security: HTTPS, CSP, XSS, CSRF, WAF, DDoS Protection

A website breach rarely looks like in movies. More often it's: a bot finds an unprotected /admin/export endpoint, downloads the customer database, and closes the connection. Or: through an outdated WordPress plugin, a web shell is uploaded, and the server starts sending spam. Or quieter: an XSS in a comment field allows stealing admin session cookies, unnoticed for months. We have analyzed dozens of such cases — each vulnerability could have been fixed at the development or audit stage.

Web application security is not a single setting. It's layers of protection, each closing a separate class of attacks. Order an audit — we'll assess the project and deliver a turnkey plan within 2–4 weeks.

How do we ensure comprehensive web application security?

HTTPS and Proper TLS Configuration

HTTPS is the minimum mandatory level. But having an SSL certificate and having a properly configured TLS are different things.

In Nginx/Apache configuration we check:

• Protocols: only TLS 1.2 and TLS 1.3, SSLv3 and TLS 1.0/1.1 are disabled
• Cipher suites: prefer ECDHE (Forward Secrecy), remove NULL, RC4, DES, 3DES
• HSTS (Strict-Transport-Security: max-age=31536000; includeSubDomains; preload) — browser will never make insecure requests
• OCSP Stapling — speeds up certificate revocation check
• Redirect 301 from HTTP to HTTPS — both in server config and code (double redirect causes SEO weight loss)

Check: SSL Labs (ssllabs.com/ssltest) should show A or A+. If B, the configuration is weak.

Let's Encrypt + Certbot for production is standard. Automatic renewal via certbot renew in cron. Wildcard certificates for subdomains via DNS-01 challenge.

Content Security Policy: The Most Powerful and Complex Protection

CSP is an HTTP header that tells the browser which sources are allowed to load resources. A properly configured CSP completely blocks most XSS attacks, even if the vulnerability exists in the code.

The problem: breaking the site with an incorrect CSP is easy. default-src 'none' — and fonts, images, JS stop working. So we start with Content-Security-Policy-Report-Only — CSP logs violations but does not block anything. We monitor reports for 2–4 weeks, refine the policy, then switch to enforcement mode.

Example of a real policy for a site with Google Analytics, Google Fonts, and Stripe:

Content-Security-Policy:
  default-src 'self';
  script-src 'self' https://www.googletagmanager.com https://js.stripe.com 'nonce-{random}';
  style-src 'self' https://fonts.googleapis.com 'unsafe-inline';
  font-src 'self' https://fonts.gstatic.com;
  frame-src https://js.stripe.com;
  img-src 'self' data: https://www.google-analytics.com;
  connect-src 'self' https://api.stripe.com https://www.google-analytics.com;
  report-uri /csp-report;

nonce — a random string generated server-side per request. Inline scripts with the correct nonce are allowed; without nonce, they are blocked. This completely breaks XSS via <script>alert(1)</script>.

'unsafe-inline' in style-src is a compromise for inline styles. It's better to remove it by moving all styles to CSS files, but that requires refactoring.

Why XSS Remains the Most Common Vulnerability?

XSS (Cross-Site Scripting) — injection of JS code through user input. According to OWASP, XSS is in the top 3 web application vulnerabilities. Three types:

Protection: never insert user input into HTML without escaping. In PHP — htmlspecialchars() with ENT_QUOTES. In Laravel Blade templates — {{ $var }} is safe, {!! $var !!} is dangerous. In React — {variable} is safe, dangerouslySetInnerHTML is dangerous. For Rich Text — use htmlpurifier on PHP or DOMPurify in the browser.

CSRF: Protecting Forms and APIs

CSRF (Cross-Site Request Forgery) — an attacker forces the victim's browser to send a request on their behalf. Example: a user is logged into a bank, opens a malicious page, which makes fetch('https://bank.ru/transfer?to=evil&amount=50000') — if the bank is unprotected, money is transferred.

CSRF tokens — standard protection for forms: the server generates a random token, stores it in the session, and inserts it as a hidden field in the form. On POST request, the token is verified. The attacker does not know the token. Laravel does this automatically with @csrf.

SameSite cookies — modern protection: SameSite=Strict or SameSite=Lax prevents the browser from sending cookies in cross-site requests. Works in all modern browsers.

API without sessions (JWT, Bearer tokens) — CSRF is irrelevant if the token is not stored in a cookie (but in the Authorization header or localStorage). However, localStorage is vulnerable to XSS — so for sensitive data, HttpOnly cookies with SameSite are preferable.

WAF and DDoS Protection

WAF (Web Application Firewall) filters HTTP traffic for attacks: SQL injection, XSS, path traversal, known exploit patterns. Options:

• Cloudflare WAF — cloud-based, OWASP Top 10 rules out of the box, custom rules via expressions. Managed Rules automatically block new threats.
• ModSecurity (Nginx/Apache) — self-hosted, OWASP Core Rule Set (CRS). Flexible but requires tuning and monitoring of false positives.
• AWS WAF — for infrastructure on AWS, integrates with CloudFront and ALB.

DDoS protection. Cloudflare at L3/L4/L7 is the de facto standard for most sites. Automatic mitigation of volumetric attacks, Under Attack Mode during active attacks. For critical infrastructure — Cloudflare Magic Transit or specialized solutions (Qrator, StormWall for the Russian market).

Rate Limiting at the application level — an additional layer. Laravel ThrottleRequests middleware: 60 requests per minute per IP for general endpoints, 5 for /login and /password/reset. Redis as a counter store — mandatory for horizontally scalable systems (otherwise limits are not synchronized between servers).

Other Mandatory Measures

Security headers. Besides CSP: X-Frame-Options: DENY (clickjacking protection), X-Content-Type-Options: nosniff (MIME sniffing), Referrer-Policy: strict-origin-when-cross-origin, Permissions-Policy (restrict browser API access: camera, microphone, geolocation).

SQL injection. Prepared statements everywhere. No concatenation of user input into SQL strings. ORM (Eloquent, Doctrine) protects by default. $wpdb->prepare() in WordPress is mandatory.

Dependency updates. composer audit and npm audit in CI/CD pipeline. Dependabot or Renovate for automatic PRs with updates. Critical CVEs — patch within 24 hours.

Secrets and configuration. .env — never in Git. Secrets in production — via CI/CD environment variables (GitHub Secrets, GitLab CI Variables) or HashiCorp Vault. Leak detection: git-secrets, truffleHog in pre-commit hooks.

How We Work

1. Audit — code scanning, configuration review, dependency analysis, manual business logic verification.
2. Planning — vulnerability remediation plan, stack selection (CSP, WAF, rate limiting).
3. Implementation — TLS setup, CSP configuration, headers, Rate Limiting, WAF.
4. Testing — re-penetration test, load testing, false positive check.
5. Deployment and Monitoring — enable production CSP, set up alerts, train the team.

What's Included

• Report with found vulnerabilities and recommendations (PDF + code snippets)
• Ready TLS configuration (Nginx/Apache)
• CSP policy with Report-Only and production versions
• WAF and Rate Limiting setup
• Dependency update plan
• Access to monitoring tools (Sentry, Datadog)
• 30 days of post-audit support (consultations, fixes)

Timeline and Cost

The budget is calculated individually — contact us for a project evaluation.