Optimizing Asynchronous Processing in Distributed Systems: Insights from Facebook and Alibaba

This article is based on Zhao Haiping’s keynote at the 2015 QCon Global Software Development Conference (Beijing), where he shared his experience optimizing backend performance at Facebook and Alibaba.

During his eight years at Facebook, Zhao focused on backend performance tuning such as PHP and memcache optimization. After moving to Alibaba, he aimed to build a comprehensive profiling system to locate local bottlenecks, followed by large‑scale architectural and JVM‑level optimizations.

The talk’s core theme is the optimization of asynchronous processing in distributed systems.

Single‑Machine Era Data Requests

Fifteen years ago, software architecture was simple: one client per DB server or many clients sharing a single DB server, with the debate of whether to embed complexity in the DB (e.g., Oracle) or in the client (e.g., PHP). Early Facebook exemplified the latter.

Big‑Data Era Data Requests

With the rapid growth of the Internet, a single DB server can no longer store all user data, and each user’s data footprint has expanded (personal info, social graph, behavior logs). This raises the question of how to write programs that query multiple DB servers.

Two approaches are discussed:

Serial synchronous queries: query DB1, obtain result, then use it to query DB2 (dependent queries).

Parallel synchronous queries: query DB1 and DB2 simultaneously when there is no dependency. Facebook introduced a helper function: ExecParallelQuery(conn1,Query1,conn2,Query2) However, parallelism introduces waiting problems when different functions, possibly written by different teams, depend on each other’s results, leading to slow page loads.

Asynchronous Processing Idea

Asynchronous processing records the DB servers a function will need to access, postpones actual execution, and returns a Future object. Example: conn.asyncExec(Query) returns a Future that can be scheduled later. By collecting Futures, the system can build a dependency tree, executing independent nodes in parallel and dependent nodes serially.

All functions must adopt the async style; otherwise, mixed return types (Future vs. raw value) break the execution flow. Converting an entire codebase to async is painful because every call chain must be updated.

After conversion, each function returns a Future, forming a tree where leaf nodes execute first, and parent nodes wait for their children. The tree is visualized as:

Each node has two states: waiting or completed. Sibling nodes run in parallel; parent‑child nodes run sequentially, ensuring correct ordering.

Remaining Challenges for Asynchronous Processing

Even with async, ordering of queries and cache placement remain open problems. For example, deciding whether to query a friend list first or a rare‑item buyer list first should be based on cardinality, not hard‑coded order. Traditional databases use INNER JOIN to let the optimizer choose the best order, but programming languages lack such mechanisms.

Thread‑based solutions are inefficient on Linux when thread counts exceed 200‑300 due to overhead.

Cache placement decisions (e.g., which class to store in memcache) also need dynamic analysis; static decisions can quickly become suboptimal.

How to Make Asynchronous Execution Better?

Query ordering and cache decisions should be handled in a separate scheduling phase rather than at coding time. No existing language fully supports this, suggesting a need for new tools or languages (perhaps lazy evaluation like Haskell) to provide systematic async support.

The speaker encourages the community to treat computing as a science, aiming for future tools that make distributed system programming as straightforward as single‑machine development.

Speaker Biography

Zhao Haiping, Alibaba Technical Assurance Researcher, began programming early, won multiple student computing contests, studied biology at Peking University, earned a molecular biology master’s at NYU, switched to computer science for a master’s at Princeton, worked at Microsoft, then joined Facebook in 2007 where he created the HipHop project that accelerated PHP 5‑6×, saving billions. After HipHop, he focused on async processing for distributed systems, adding yield and generator support to PHP. He returned to China in 2015 to tackle performance challenges at Alibaba.

Disclaimer: The content is sourced from public channels, presented neutrally for reference and discussion only. Copyright belongs to the original authors or institutions; please contact for removal if infringed.