Hermetic Servers for Reliable End-to-End Testing

Imagine a complex rich web application backed by a network of interacting server clusters, where each user action traverses many service calls from the click to data storage.

Google’s large-scale apps such as Gmail and Google+ face the same situation, raising the question of how to write reliable end-to-end tests for them.

End-to-End Testing

In Google’s testing terminology, an end-to-end test validates the entire service stack, from the initial user request to the final response.

The diagram below illustrates a typical end-to-end test flow. Note that the front-end server under test may call a third‑party back-end service that the simulated user does not need to request.

Writing fast and reliable automated end-to-end tests for such systems is challenging, especially when network access must be avoided; network-dependent tests tend to be slower and flaky.

Hermetic Test Environment

Google’s technique for designing end-to-end tests is to use Hermetic Servers , which provide a sealed test environment.

A hermetic server means that all external dependencies required by a test case run on the same node, eliminating the need for network access and ensuring consistent behavior.

Although “sealed” can refer to complete isolation, in this context the hermetic server runs on a single machine, which is a minimal form of sealing that is crucial for tests.

Why hermetic servers are beneficial?

If the system under test consists of hermetic servers, it can be started entirely on one server—physical or virtual—without any network calls.

Design of Hermetic Servers

When designing a hermetic server, consider the following:

Inject all external service connections at runtime (e.g., via command-line flags or Guice).

Bundle required static files with the service binary.

If the server accesses storage, replace it with file-based or in-memory fakes.

These practices give the server a high degree of configurability, making it a candidate for hermetic use.

Additional steps are needed before the server is test-ready:

Ensure the test does not use real connection points, using appropriate fakes or mocks instead.

Provide modules that simplify assembling test data for storage.

Provide logging modules to trace request/response chains within the system.

Using Hermetic Servers in Tests

Consider the example diagram: all services are hermetic. The end-to-end test proceeds as follows:

Start the system under test.

Issue a request from a test client.

Verify the response returned by the system.

The test does not need to connect to the simulated back-end server. If a test does require that back-end, a hermetic version must be supplied at the connection point.

This approach yields more reliable tests because no network is used, and it is faster because everything resides in memory or on local disk.

Google runs these tests in its continuous‑integration system, so any code change to a service triggers the tests, and log modules help locate failures.

Common scenarios where Google employs hermetic servers include:

Guice‑based server startup tests to catch injection errors.

API tests for back-end services.

Mini performance tests.

UI and API tests for front-end services.

Conclusion

Hermetic servers have limitations: they increase test runtime because the entire system must be started for each test, and resource-constrained environments may hit memory or CPU limits as interactions grow. Moreover, in-memory data sets are smaller than production data.

Nevertheless, hermetic servers are a valuable testing tool that, like any tool, should be used thoughtfully where appropriate.

Published: October 03, 2012

Original Authors: Chaitali Narla and Diego Salas

Original Link: Hermetic Servers