How WeChat Scales to Millions with libco: A Deep Dive into C/C++ Coroutines

libco is a C/C++ coroutine library that has been used at massive scale in WeChat’s backend since 2013, running on tens of thousands of machines. The project was first open‑sourced in 2013 and recently received a major update, available at https://github.com/tencent/libco . libco supports agile synchronous‑style programming while providing high concurrency capabilities.

Features of libco

Supports CGI framework for easy web service construction (New).

Integrates common third‑party libraries such as gethostbyname, mysqlclient, and SSL (New).

Optional shared‑stack mode enables millions of connections on a single machine (New).

Simple coroutine programming interface:

pthread‑like API with functions like co_create and co_resume.

Coroutine‑private variables and semaphore co_signal (New).

Non‑language‑level lambda support for writing asynchronous tasks inline (New).

Lightweight network framework based on epoll/kqueue and a high‑performance timer wheel.

Background of libco

Early WeChat backend modules used a hybrid semi‑synchronous, semi‑asynchronous model: the access layer was asynchronous, while business logic ran in synchronous multi‑process or multi‑thread models, limiting concurrency to a few hundred. As the business grew, the system scale expanded, making services vulnerable to backend or network jitter.

Choosing Asynchronous Refactoring

Converting all services to a fully asynchronous model would require massive engineering effort and carry high risk. Therefore, the team considered using coroutines as a less invasive solution.

Challenges of Using Coroutines

Adopting coroutines in C/C++ presented several challenges:

Lack of large‑scale industry experience with C/C++ coroutines.

Controlling coroutine scheduling.

Handling synchronous‑style API calls such as sockets and mysqlclient.

Managing existing global variables and thread‑local storage.

libco’s Solution

libco addressed all the above challenges, enabling non‑intrusive asynchronous refactoring of hundreds of WeChat backend modules with minimal changes to business logic. Today, most services run on a multi‑process or multi‑thread coroutine model, dramatically improving concurrency.

libco Framework

Handling Synchronous‑Style APIs

Synchronous network calls (connect, write, read, etc.) block the thread while waiting for I/O. libco hooks these calls, registers the network operation as an asynchronous event, yields the coroutine, and resumes it when the event completes, preserving the clear logic of synchronous code while achieving high concurrency.

Supporting Ten‑Million‑Level Coroutines

By default each coroutine owns its own stack, but libco also offers a stackless shared‑stack mode where multiple coroutines share a single stack, copying stack contents only on context switches. This reduces memory overhead and enables millions of concurrent connections.

Using the shared‑stack mode, libco created ten million coroutines on a server (E5‑2670 v3 @ 2.30GHz ×2, 128 GB RAM), consuming about 66 GB of memory for 1 million coroutines sharing 128 KB each.

Coroutine Private Variables

libco introduces ROUTINE_VAR, a coroutine‑local variable that behaves like thread‑local storage in non‑coroutine environments but is isolated per coroutine when running under libco, eliminating re‑entrancy issues and simplifying migration.

Hooking gethostbyname

To make DNS resolution non‑blocking, libco hooks the glibc gethostbyname implementation by intercepting the internal __poll call and using coroutine‑private variables, achieving asynchronous DNS without modifying glibc source.

Coroutine Semaphore

libco provides co_signal, a coroutine semaphore, with co_cond_signal and co_cond_broadcast for notifying one or all waiting coroutines, analogous to pthread signals in multithreaded environments.

Summary

libco is an efficient C/C++ coroutine library offering a complete programming interface, socket‑family function hooks, and coroutine‑private variables, enabling rapid development with a synchronous programming model. After years of stable operation, libco has become a cornerstone of the WeChat backend framework.