Mastering Varnish: Architecture, VCL Functions, and Configuration Guide

Varnish is a high‑performance open‑source HTTP accelerator (a caching reverse proxy) that stores responses in memory or files to speed up web servers. Compared with Squid, Varnish offers higher performance, faster speed and easier management.

Varnish Program Structure

Management process: compiles VCL, applies new configuration, monitors Varnish, initializes Varnish, provides a CLI, initializes cache memory structures and partitions memory into blocks of various sizes.

Child/Cache threads:

Acceptor: accepts new connections;

Worker: processes and responds to user requests;

Expiry: cleans expired cache objects.

Log: Shared Memory Log, typically 90 MB, divided into counters and request‑related data.

Cache data management:

Expires: defines cache lifetime;

http validation;

If-Modified-Since/Last-Modified: Varnish asks the backend if the object has changed since cached; returns 304 if unchanged.

If-None-Match/Etag: uses a tag and timestamp to validate cached content.

VCL built‑in functions and processing flow (state engine)

VCL built‑in functions: vcl_recv, vcl_fetch, vcl_pipe, vcl_hash, vcl_pass, vcl_hit, vcl_miss, vcl_deliver, vcl_error.

vcl_recv: handles incoming request, decides routing, backend selection, etc.

vcl_fetch: decides whether to store response in cache or pass it directly.

vcl_pipe: pipes unknown requests directly to backend.

vcl_hash: defines data used for hash key generation.

vcl_pass: forwards request to backend without caching.

vcl_hit: actions when cache object is found.

vcl_miss: actions when cache object is not found.

vcl_deliver: prepares response for client.

vcl_error: synthesizes error responses.

VCL syntax:

//, #, /* */ comment.

sub $NAME defines a function.

No loops supported.

Many built‑in variables.

Supports termination statements without return values.

Domain‑specific language.

Operators: =, ==, ~, !, &&, ||.

Common variables:

Global: .Now, .host, .port.

Request phase: client.ip, server.hostname, server.ip, server.port, req.request, req.url, req.proto, req.backend, req.backend.healthy, req.http.HEADER, req.can_gzip, req.restarts.

Before backend request: bereq.request, bereq.url, bereq.proto, bereq.http.HEADER, bereq.connect_timeout.

After backend response, before caching: beresp.backend.ip, beresp.ttl, beresp.do_stream, beresp.do_gzip, beresp.do_gunzip, beresp.http.HEADER, beresp.proto, beresp.status, beresp.response, beresp.backend.name, beresp.backend.port, beresp.storage.

Cache object: obj.proto, obj.status, obj.response, obj.ttl, obj.hits, obj.http.HEADER.

Hash calculation: req.hash.

Client response: resp.http.HEADER, resp.proto, resp.status, resp.response.

Varnish configuration example:

/etc/sysconfig/varnish defines runtime characteristics.

/etc/varnish/default.vcl defines default VCL state engine.