Inside Facebook’s ‘Hotfix Bar’: Secrets of Massive Deployments

Facebook’s headquarters in Menlo Park, once the home of Sun Microsystems, features a "zan" (thumb‑up) sign that visitors love to photograph. The author was granted exclusive access to explore the campus, interview the release engineering team, and see first‑hand the technologies that keep Facebook handling billions of user requests daily.

HipHop Optimizer

Most of Facebook’s source code is written in PHP, a language known for rapid development but slower execution. To improve performance, Facebook created HipHop, a tool that translates PHP into highly optimized C++ code, which is then compiled into native binaries. Open‑sourced in 2010, HipHop cut Facebook’s CPU consumption by about 50%.

BitTorrent Deployment System

Compiled binaries for the entire codebase are roughly 1.5 GB. Distributing this massive file to thousands of servers is a major challenge, so Facebook built a custom BitTorrent tracker. Servers pull pieces of the binary from peers in the same rack or node, dramatically reducing transfer time. An average full deployment takes about 30 minutes—15 minutes to compile and 15 minutes to push the binary via BitTorrent.

Testing and Continuous Integration

Facebook runs small updates daily and larger ones weekly. Developers test new code on internal servers (the “A2” layer) before it reaches the broader user base. Automated testing tools perform both unit tests and user‑interaction simulations. An internal bug‑reporting tool lets engineers quickly flag issues.

Preparation and Coordination

The release team uses an internal IRC system with bots to coordinate updates. Before a deployment, a “check‑in” message is sent; developers must acknowledge readiness. If a response is missing, the bot contacts the engineer via email or SMS, ensuring everyone is prepared.

Release Process

During a deployment, a web‑based dashboard shows a progress bar and highlights any servers that fail to receive the update. Facebook’s architecture is stateless and distributed, allowing servers to continue serving traffic while being updated. The process is non‑interruptive; the site never goes into maintenance mode.

Post‑Update Verification

After deployment, the team monitors metrics such as traffic, resource usage, and error rates. Custom monitoring tools compare current data with historical baselines to quickly identify anomalies. If a serious bug is discovered, the team works with the responsible developer to fix it and roll out a new version.

Rollback Philosophy

Facebook rarely rolls back; the team believes “only losers roll back.” Binaries from the previous version are kept as a safety net, but rollbacks are used only in extreme cases.

Future Directions

Facebook is developing a new HipHop Virtual Machine that will compile PHP to bytecode, reducing deployment size from gigabytes to small bytecode patches. This will enable near‑instant updates and more incremental deployment, further accelerating the development cycle.