Custom Varnish VCL rules development

Developing Custom Varnish VCL Rules

Varnish Cache without custom VCL is a hammer trying to drive screws. Default configuration caches everything or nothing, ignores auth headers, can't handle mobile site versions, and crashes on the first edge case with cookies.

Custom VCL rules solve specific tasks: cache management by conditions, request normalization, CDN bypass for certain routes, routing to different backends, grace mode when origin is down.

VCL Architecture and Entry Points

VCL (Varnish Configuration Language) is a domain-specific language with a state machine: vcl_recv, vcl_hash, vcl_hit, vcl_miss, vcl_pass, vcl_fetch (Varnish 4+: vcl_backend_fetch/vcl_backend_response), vcl_deliver.

Typical starting point for customization—vcl_recv. Here we decide what to cache, what to bypass, what to serve from cache without hitting the backend.

vcl 4.1;

import std;
import directors;

backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .connect_timeout = 2s;
    .first_byte_timeout = 60s;
    .between_bytes_timeout = 10s;
    .probe = {
        .url = "/healthz";
        .timeout = 1s;
        .interval = 5s;
        .window = 5;
        .threshold = 3;
    }
}

backend api {
    .host = "127.0.0.1";
    .port = "8081";
    .connect_timeout = 1s;
    .first_byte_timeout = 30s;
}

Request Normalization

Without normalization, the same resource is stored under dozens of keys: /?utm_source=google, /?utm_source=facebook, /?fbclid=abc123—different cache entries for identical content.

sub vcl_recv {
    # Remove marketing parameters before building cache key
    if (req.url ~ "(\?|&)(utm_source|utm_medium|utm_campaign|utm_term|utm_content|fbclid|gclid|yclid|_ga|mc_eid)=") {
        set req.url = regsuball(req.url, "&(utm_source|utm_medium|utm_campaign|utm_term|utm_content|fbclid|gclid|yclid|_ga|mc_eid)=[^&]*", "");
        set req.url = regsuball(req.url, "\?(utm_source|utm_medium|utm_campaign|utm_term|utm_content|fbclid|gclid|yclid|_ga|mc_eid)=[^&]*&?", "?");
        set req.url = regsub(req.url, "\?$", "");
    }

    # Normalize Accept-Encoding—Varnish stores separate objects for gzip/br/identity
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|jpeg|png|gif|webp|gz|tgz|bz2|tbz|mp3|ogg|swf|flv|mp4|woff2?)$") {
            unset req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "br") {
            set req.http.Accept-Encoding = "br";
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } else {
            unset req.http.Accept-Encoding;
        }
    }

    # Normalize cookies—keep only needed for auth
    if (req.http.Cookie) {
        set req.http.Cookie = ";" + req.http.Cookie;
        set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
        set req.http.Cookie = regsuball(req.http.Cookie, ";(session|auth_token|XSRF-TOKEN)=", "; \1=");
        set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");
        if (req.http.Cookie == "") {
            unset req.http.Cookie;
        }
    }
}

Routing by Request Type

API and static assets need different logic. API—short TTL or pass, static—long TTL with normalization.

sub vcl_recv {
    # API—don't cache, pass directly
    if (req.url ~ "^/api/") {
        return(pass);
    }

    # Route to separate backend for API
    if (req.url ~ "^/api/v[0-9]+/") {
        set req.backend_hint = api;
        return(pass);
    }

    # Authorized users—personal content, don't cache
    if (req.http.Authorization || req.http.Cookie ~ "auth_token=") {
        return(pass);
    }

    # POST, PUT, DELETE, PATCH—don't cache
    if (req.method != "GET" && req.method != "HEAD") {
        return(pass);
    }

    # Static—cache aggressively, remove cookies
    if (req.url ~ "\.(css|js|jpg|jpeg|png|gif|ico|svg|woff|woff2|ttf|eot|webp|avif)(\?.*)?$") {
        unset req.http.Cookie;
        return(hash);
    }

    return(hash);
}

Grace Mode and Stale-While-Revalidate

Grace allows serving stale cache while backend is overloaded or temporarily down. Critical for high-load sites.

sub vcl_backend_response {
    # Set grace period: 24 hours after TTL expiration
    set beresp.grace = 24h;

    # If backend returned 5xx—save object for grace
    if (beresp.status >= 500) {
        set beresp.ttl = 0s;
        set beresp.grace = 60s;
        return(deliver);
    }

    # Custom TTL by content type
    if (bereq.url ~ "^/news/") {
        set beresp.ttl = 10m;
    } elsif (bereq.url ~ "^/static/") {
        set beresp.ttl = 30d;
        unset beresp.http.Set-Cookie;
    } elsif (bereq.url ~ "^/product/") {
        set beresp.ttl = 1h;
    } else {
        set beresp.ttl = 5m;
    }

    # Don't cache if backend explicitly forbids
    if (beresp.http.Cache-Control ~ "no-store|private") {
        set beresp.uncacheable = true;
        return(deliver);
    }
}

sub vcl_hit {
    # If object is stale but grace allows—serve and revalidate background
    if (obj.ttl >= 0s) {
        return(deliver);
    }
    if (obj.ttl + obj.grace > 0s) {
        return(deliver);
    }
    return(restart);
}

Cache Invalidation

Purge by tags (via xkey/surrogate keys)—the right approach for CMS with object dependencies.

import xkey;

sub vcl_recv {
    # Simple purge by URL
    if (req.method == "PURGE") {
        if (!client.ip ~ purge_acl) {
            return(synth(405, "Not allowed"));
        }
        return(purge);
    }

    # Soft-purge by tag (xkey)
    if (req.method == "XKEY-PURGE") {
        if (!client.ip ~ purge_acl) {
            return(synth(405, "Not allowed"));
        }
        set req.http.n-gone = xkey.softpurge(req.http.xkey-purge);
        return(synth(200, "Purged " + req.http.n-gone + " objects"));
    }
}

sub vcl_backend_response {
    # Backend sends surrogate keys in header
    if (beresp.http.Surrogate-Key) {
        set beresp.http.xkey = beresp.http.Surrogate-Key;
    }
}

Invalidation call from application:

# Purge specific page
curl -X PURGE https://example.com/news/article-123

# Purge by tag (all pages related to category)
curl -X XKEY-PURGE -H "xkey-purge: category:5" https://example.com/

Device Variants

If mobile version is served from the same URL, store separate objects for desktop and mobile.

sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }
    # Add device type to cache key
    if (req.http.X-Device-Type) {
        hash_data(req.http.X-Device-Type);
    }
    return(lookup);
}

sub vcl_recv {
    # Detect device type by User-Agent
    if (req.http.User-Agent ~ "(?i)mobile|android|iphone|ipod|blackberry|opera mini|iemobile") {
        set req.http.X-Device-Type = "mobile";
    } else {
        set req.http.X-Device-Type = "desktop";
    }
}

Debugging and Metrics

sub vcl_deliver {
    # Add debug headers (remove in prod or restrict by IP)
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
        set resp.http.X-Cache-Hits = obj.hits;
    } else {
        set resp.http.X-Cache = "MISS";
    }
    set resp.http.X-Served-By = server.hostname;
    # Remove internal headers
    unset resp.http.X-Powered-By;
    unset resp.http.Server;
    unset resp.http.X-Varnish;
    unset resp.http.Via;
}

Monitoring via varnishstat and varnishlog:

# Current hit rate
varnishstat -f MAIN.cache_hit,MAIN.cache_miss

# Live log filtered by URL
varnishlog -q 'ReqURL ~ "^/news/"' -g request

# Top cache miss by URL
varnishtop -i ReqURL -q 'VCL_call eq "MISS"'

Typical Project Timeline

Day 1–2—traffic audit, analyze backend response headers, identify non-cacheable patterns (cookies, Cache-Control: private).

Day 3–4—write basic VCL: URL normalization, strip marketing parameters, separate static and dynamic.

Day 5—configure grace, health checks, test on staging with varnishtest.

Day 6—implement invalidation via PURGE/xkey, integrate with CMS or deploy pipeline.

Day 7—load testing, configure hit rate monitoring, final deploy.

Complex cases (A/B testing via Varnish, ESI includes, multi-level caching with Nginx) add 3–5 days.