Autonomous Integration Testing Infrastructure at Facebook: Design, Challenges, and Practices

Rapid prototyping, testing, and iteration are essential for high‑quality software delivery, but they require a stable infrastructure that minimizes unnecessary friction.

Two approaches are proposed: better abstraction of services and automation of tests.

1. Defining the Test Environment

Integration tests run in dedicated, deterministic environments separate from production to avoid side effects. Unlike unit tests, integration tests involve multiple services and rely less on mocks, using shadow instances of production services with read‑only access and optional isolation layers.

Facebook reuses its production container and routing infrastructure to create temporary test entities, allowing tests to interact with production‑like services safely.

2. Test Input Sources

Test fixtures execute services directly or modify the test environment, while mocks can provide predefined responses. Fuzzing generates random inputs that conform to service contracts, and Facebook leverages Thrift’s reflection to automatically construct inputs and mock dependencies.

Record‑and‑playback captures real production traffic, mutates it, and replays it in tests, providing realistic inputs without requiring a separate test harness.

3. Test Assertions

Assertions focus on externally observable behavior such as RPC responses, mock call parameters, and data written to temporary databases. The infrastructure also detects crashes, health‑check failures, and unexpected logs.

4. Scalability and Extensibility

The platform allows teams to extend the infrastructure for common patterns (e.g., test environment setup) or specialized tests like disaster‑recovery scenarios. Isolation can be implemented at the application level (via API filtering) or at the network level using IP:PORT filtering, with Facebook choosing LD_PRELOAD for flexibility.

Facebook’s autonomous testing deployment follows a two‑stage strategy: initially running tests silently in the background to gather data, then encouraging opt‑in execution before service deployment. By October 2021, the system fuzzed roughly one‑third of Thrift services, uncovering over a thousand bugs and providing detailed reports to service owners.

Key takeaways include the need for fine‑grained read‑only API marking, richer test‑environment abstractions, better bug‑diagnosis information, and metrics to assess fuzzing effectiveness and coverage.