How Airbnb Built a Scalable, Reliable Kubernetes Sidecar for Dynamic Configuration

Configuration distribution lifecycle

Step 1 – Create/Update config : Developers commit changes via Git or a web UI. The Sitar Service stores a full version history and enforces ACLs.

Step 2 – Hourly snapshot upload : Snapshot Service packages the complete state of all config groups and uploads a compressed snapshot to AWS S3.

Step 3.1 – Pre‑load snapshot from S3 (pod start) : When a pod starts, the sitar‑agent sidecar downloads the latest snapshot for each tenant from S3 and writes it to a shared volume. This provides a fast, reliable bootstrap and isolates the pod from temporary Sitar Service outages.

Step 3.2 – Pre‑load latest config from Sitar Service (pod start) : After the snapshot is mounted, the sidecar performs an initial sync with the Sitar Service to fetch any changes that occurred after the snapshot. On success the sidecar signals readiness, allowing the main container to start.

Step 4 – Periodic updates : After startup the sidecar enters a second‑level polling loop with jitter. Each cycle it queries the Sitar Service for changes to all subscribed config groups.

Step 5 – Read config : The main container reads the files via the Sitar client library, which maintains an in‑memory cache that refreshes transparently on file changes.

Sidecar vs. in‑process deployment

Advantages of moving the agent into the main container

Eliminates a JVM per pod, reducing cost and sharing memory/CPU.

Reduces operational surface by removing one container.

Disadvantages of moving the agent into the main container

Requires implementing the sidecar logic in every supported language (Java, Python, Go, TypeScript, Ruby), increasing development and maintenance effort.

Loss of isolation: bugs or resource spikes in the agent could crash the main application and vice‑versa.

Logs and metrics become intermingled, making debugging harder.

Optimising the agent independently becomes more difficult.

Because reliability and multi‑language maintenance costs outweighed the cost savings, the agent remains an isolated sidecar container.

Pull model and server‑side optimisations

The agent polls the Sitar Service every 10 seconds. To reduce load when no changes occur, a short TTL (10 s) cache sits in front of the service; most requests hit the cache and avoid expensive database work. On a cache miss the request carries a token indicating the last scanned DB row, allowing the service to skip rows already processed and further reduce DB load.

Configuration edits are manual and take several seconds, so a few‑second propagation delay is acceptable. The simple, stateless pull architecture is retained for its operational advantages.

Local data‑store selection

The original store used Sparkey, which is write‑once/read‑many, lacks native concurrency, requires whole‑file locking, rebuilds indexes on every write, and has limited multi‑language bindings. Two alternatives were benchmarked: SQLite and RocksDB.

SQLite

Mature, widely used, official bindings for Java, Node.js/TypeScript, Python, Go, Ruby.

Built‑in WAL provides concurrent reads during writes.

Simple operations – single file, no background compaction.

Read/write performance far exceeds Sparkey and meets workload needs.

RocksDB

Best raw read/write performance; stable at 1500 ops/sec with minimal degradation.

Read latency 2–3× higher than SQLite and grows linearly with data size.

Requires tuning of compaction, block cache, column families, making operations more complex.

Multi‑language ecosystem less mature.

Both RocksDB and SQLite outperformed Sparkey across data‑size, memory‑allocation, and read‑QPS dimensions. Although RocksDB offered superior raw performance, SQLite’s multi‑language support, WAL‑based concurrency, and simpler operational model made it the overall better fit. The team migrated from Sparkey to SQLite.

Safe migration strategy

Shadow reads : Before switching a service, read results from Sparkey and SQLite in parallel and compare them to ensure parity.

Feature‑flag‑driven progressive rollout : Start migration with low‑impact services and incrementally move to Tier‑0 services, coordinating at each step.

Conclusion

Sitar‑agent is the core of Airbnb’s dynamic‑configuration system. Key trade‑offs during the Java rewrite included cost vs. isolation, simplicity vs. push‑model efficiency, and raw performance vs. operational practicality. All decisions were driven by three constraints: configuration must always be readable, changes must propagate to tens of thousands of pods within seconds, and the solution must support a heterogeneous, multi‑language service stack without adding undue maintenance burden.

Code example

步骤 3.1 — 从 S3 预加载快照 (pod 启动时)
生产服务 pod 启动时，sitar-agent sidecar 会首先运行。它从 S3 下载所订阅的各租户的最新配置快照，并保存到挂载磁盘中 (该磁盘由 sitar-agent 与主容器共享) 。这样，agent 就能从一个已知正常的状态完成引导，而不必在每次重启时都从 Sitar Service 重新获取所有配置。从 S3 预加载快照可以加快重启速度，让服务能够抵御 Sitar Service 的短暂不可用，并避免部署期间出现负载尖峰。
步骤 3.2 — 从 Sitar Service 预加载最新配置 (pod 启动时)