Coroutines and Their Implementation in Tars C++

This article introduces the concept of coroutines and provides an in‑depth analysis of the coroutine implementation in the Tars C++ framework (v3.0.0). Tars is an open‑source high‑performance RPC framework maintained by the Linux Foundation, supporting multiple languages such as C++, Java, PHP, Node.js, and Go.

Coroutines are functions that can be suspended (yield) and resumed, allowing asynchronous code to be written in a sequential style. The article reviews the historical origin of coroutines, their advantages (simplified asynchronous programming and reduced context‑switch overhead), and compares them with processes and threads.

Coroutines are classified by control‑transfer mechanism (symmetric vs. asymmetric) and by stack usage (stackful vs. stackless). Stackful coroutines have independent stacks, while stackless coroutines share a single stack and are typically implemented with state machines or closures.

Implementation of Tars coroutines

The core of Tars coroutine support relies on two key components:

User‑mode context switching.

Coroutine scheduling.

The user‑mode context switch is built on boost.context , which provides two C‑style interfaces: make_fcontext and jump_fcontext . The relevant declarations are:

/**
 * @brief Execution environment context
 */
typedef void* fcontext_t;

/**
 * @brief Event parameter wrapper
 */
struct transfer_t {
    fcontext_t fctx; // source execution context
    void* data;    // user‑provided pointer
};

/**
 * @brief Initialize execution environment context
 * @param sp   Stack address
 * @param size Stack size
 * @param fn   Entry function
 * @return Initialized fcontext_t
 */
extern "C" fcontext_t make_fcontext(void* stack, std::size_t stack_size, void (*fn)(transfer_t));

/**
 * @brief Jump to target context
 * @param to Target context
 * @param vp Parameter passed to target (stored in transfer_t::data)
 * @return Source context
 */
extern "C" transfer_t jump_fcontext(fcontext_t const to, void* vp);

The Tars coroutine classes are:

TC_CoroutineInfo – wraps the boost.context interfaces and represents a single coroutine.

TC_CoroutineScheduler – manages coroutine lifecycles and scheduling.

TC_Coroutine – inherits from the thread class for easy usage.

Key source snippets:

TC_CoroutineInfo::registerFunc (creates a coroutine and performs the initial context switch):

void TC_CoroutineInfo::registerFunc(const std::function
& callback) {
    _callback = callback;
    _init_func.coroFunc = TC_CoroutineInfo::corotineProc;
    _init_func.args = this;

    fcontext_t ctx = make_fcontext(_stack_ctx.sp, _stack_ctx.size,
                                TC_CoroutineInfo::corotineEntry); // create coroutine
    transfer_t tf = jump_fcontext(ctx, this); // switch context
    this->setCtx(tf.fctx);
}

void TC_CoroutineInfo::corotineEntry(transfer_t tf) {
    TC_CoroutineInfo* coro = static_cast
(tf.data);
    auto func = coro->_init_func.coroFunc;
    void* args = coro->_init_func.args;
    transfer_t t = jump_fcontext(tf.fctx, nullptr);
    coro->_scheduler->setMainCtx(t.fctx);
    func(args, t);
}

TC_CoroutineScheduler::switchCoro (switches execution to another coroutine):

void TC_CoroutineScheduler::switchCoro(TC_CoroutineInfo* to) {
    _currentCoro = to;
    transfer_t t = jump_fcontext(to->getCtx(), nullptr);
    to->setCtx(t.fctx);
}

The scheduler maintains five coroutine states (FREE, ACTIVE, AVAIL, INACTIVE, TIMEOUT) defined as:

enum CORO_STATUS {
    CORO_FREE = 0,
    CORO_ACTIVE = 1,
    CORO_AVAIL = 2,
    CORO_INACTIVE = 3,
    CORO_TIMEOUT = 4
};

During initialization ( TC_CoroutineScheduler::init ), a pool of TC_CoroutineInfo objects is created, each with its own stack allocated via stack_traits::allocate . The scheduler links coroutines into separate linked lists for each state, enabling fast state transitions.

void TC_CoroutineScheduler::init() {
    // allocate coroutine pool
    createCoroutineInfo(_poolSize);
    TC_CoroutineInfo::CoroutineHeadInit(&_active);
    TC_CoroutineInfo::CoroutineHeadInit(&_avail);
    TC_CoroutineInfo::CoroutineHeadInit(&_inactive);
    TC_CoroutineInfo::CoroutineHeadInit(&_timeout);
    TC_CoroutineInfo::CoroutineHeadInit(&_free);
    // ... allocate stacks and construct TC_CoroutineInfo objects ...
    _mainCoro.setUid(0);
    _mainCoro.setStatus(TC_CoroutineInfo::CORO_FREE);
    _currentCoro = &_mainCoro;
}

The main run loop ( TC_CoroutineScheduler::run ) integrates an epoll based event loop with coroutine activation. It processes epoll events, wakes up coroutines that are ready, and executes a limited number of active coroutines per iteration to avoid CPU starvation.

void TC_CoroutineScheduler::run() {
    while (!_epoller->isTerminate()) {
        if (_activeCoroQueue.empty() && TC_CoroutineInfo::CoroutineHeadEmpty(&_avail) && TC_CoroutineInfo::CoroutineHeadEmpty(&_active)) {
            _epoller->done(1000); // epoll_wait with 1‑second timeout
        }
        wakeup();
        wakeupbytimeout();
        wakeupbyself();
        int iLoop = 100;
        while (iLoop > 0 && !TC_CoroutineInfo::CoroutineHeadEmpty(&_active)) {
            TC_CoroutineInfo* coro = _active._next;
            switchCoro(coro);
            --iLoop;
        }
        if (!TC_CoroutineInfo::CoroutineHeadEmpty(&_avail)) {
            TC_CoroutineInfo* coro = _avail._next;
            switchCoro(coro);
        }
    }
}

The scheduler provides four public operations for coroutine control:

go() – start a coroutine.

yield() – voluntarily suspend the current coroutine (with optional automatic re‑activation).

sleep(ms) – suspend for a specified timeout.

put() – place a coroutine into the ready queue for immediate scheduling.

In summary, the article explains coroutine fundamentals, classifies coroutine types, and details how Tars C++ leverages boost.context to implement stackful, symmetric coroutines with an epoll‑driven scheduler. The provided source excerpts illustrate the concrete mechanisms for context creation, switching, and lifecycle management.