Mastering Web Caching with Varnish: From Basics to Advanced Configuration

What Is Web Cache?

A web cache stores copies of web resources (HTML pages, images, JavaScript, data, etc.) between the web server and the client (browser). When a request for the same URL arrives, the cache decides whether to serve the stored copy or forward the request to the origin server.

Cache Hit Rate Types

Document hit rate – measured by the number of cached documents.

Byte hit rate – measured by the amount of data served from the cache.

Cache Processing Flow

Accept request from the client.

Parse request headers.

Lookup cache for a matching entry.

Freshness check – verify that cached data is still valid.

Create response message if the cache entry is valid.

Send response to the client.

Log the transaction.

Freshness‑Check Methods

Expiration date or validity:

HTTP/1.0 – use Expires header for an absolute expiration time (e.g., Expires: Sat, 18 Jul 2015 03:41:04 GMT).

HTTP/1.1 – use Cache‑Control: max‑age=1d for a relative lifetime.

Server revalidation – the server responds with 304 Not Modified if the content has not changed, 200 OK if it has, or 404 Not Found if the original resource no longer exists.

Conditional request headers:

If‑Modified‑Since

If‑None‑Match

(ETag)

Cache‑Control Directives

Server‑side directives (sent in response headers): no-store – do not cache. no-cache – cache but revalidate before use. must-revalidate – same as no-cache. max-age – maximum age. Expires – absolute expiration time.

Client‑side directives (sent in request headers): max‑stale – accept stale responses. max‑stale=<s> – specify maximum staleness. min‑fresh=<s> – require a minimum freshness. max‑age=<s> – maximum acceptable age.

Note: Resources containing private, authentication data, or cookies should generally not be cached.

Varnish Overview

Varnish is an open‑source, high‑performance HTTP reverse‑proxy cache.

Varnish Threads

Varnish has two types of threads:

Management thread – reads configuration, allocates storage (malloc, temporary, or persistent), creates cache files, initializes management structures, forks and monitors child processes.

Child/cache threads – map storage files into memory, create and initialize free object structures.

Varnish State Machines

Varnish processes requests through nine state engines (shown in the original diagram).

VCL Configuration

VCL (Varnish Configuration Language) is a domain‑specific language for defining request handling. It supports arithmetic, logical operators, regular expressions, variable assignment ( set / unset), if statements, built‑in functions, and the return statement.

Built‑in functions include: regsub(str, regexp, sub) – replace first match. regsuball(str, regexp, sub) – replace all matches. hash_data(str) – compute a hash. purge – remove an object from the cache. return(x) – specify the next state.

Common built‑in variables: req – client request (e.g., req.url, req.method). resp – response generated by Varnish. bereq – request sent from Varnish to the backend. beresp – response received from the backend. obj – cached object.

Installation and Service Configuration

# yum -y install varnish

Key configuration parameters (found in /etc/sysconfig/varnish):

NFILES=131072               # maximum open files</code>
<code>MEMLOCK=82000               # memory for log storage</code>
<code>VARNISH_LISTEN_PORT=80      # listening port (default 6081)</code>
<code>VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1</code>
<code>VARNISH_ADMIN_LISTEN_PORT=6082</code>
<code>VARNISH_MIN_THREADS=50</code>
<code>VARNISH_MAX_THREADS=1000</code>
<code>VARNISH_THREAD_TIMEOUT=120</code>
<code>VARNISH_STORAGE_FILE=/var/lib/varnish/varnish_storage.bin</code>
<code>VARNISH_STORAGE_SIZE=1G</code>
<code>VARNISH_TTL=120            # default TTL in seconds</code>
<code>DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
    -f ${VARNISH_VCL_CONF} \
    -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
    -t ${VARNISH_TTL} \
    -w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \
    -u varnish -g varnish \
    -S ${VARNISH_SECRET_FILE} \
    -s ${VARNISH_STORAGE}"

VCL Example: Adding X‑Cache Header

backend default {
    .host = "172.16.2.14";
    .port = "80";
}

sub vcl_deliver {
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
    } else {
        set resp.http.X-Cache = "MISS";
    }
    return (deliver);
}

Setting Cache Duration

sub vcl_fetch {
    if (bereq.http.Set-Cookie) {
        unset bereq.http.Set-Cookie;
        set beresp.ttl = 120s;
    }
    return (deliver);
}

Using PURGE to Clear Cache

acl clean {
    "127.0.0.1";
    "172.16.2.0"/24;
}

sub vcl_recv {
    if (req.method == "PURGE") {
        if (!client.ip ~ clean) {
            return (synth(405, "Not allowed"));
        }
        return (lookup);
    }
}

sub vcl_hit {
    if (req.method == "PURGE") {
        purge;
        return (synth(200, "Purged"));
    }
}

sub vcl_miss {
    if (req.method == "PURGE") {
        return (synth(200, "no cache"));
    }
}

Testing the Configuration

After loading the VCL file with varnishadm vcl.load t1 /etc/varnish/test.vcl and activating it with varnishadm vcl.use t1, you can verify cache hits with normal requests and clear the cache using curl -X PURGE http://your‑host. The response headers will show X-Cache: HIT or X-Cache: MISS accordingly.

Related Links

Cache‑Control (Wikipedia)

Varnish Cache Documentation