Hunting a systemd Event Loop Bug: From vmcore to Upstream Fix

On-Site Analysis: Pending Queue Non-Empty but PID 1 Blocked in epoll_wait

During a TencentOS Server 4 boot, the log stopped at Mounted sysroot. The target rootfs was mounted, but systemd did not proceed to switch-root. A vmcore dump showed all CPUs idle with no kernel deadlock or I/O block. Using crash and crash-gcore, the team extracted PID 1's core dump and analyzed it with GDB. The run queue and pending queue in sd-event contained pending tasks (including initrd-root-fs.target and dracut-pre-pivot.service), yet PID 1 was stuck in an indefinite epoll_wait(-1). The key invariant: when user-space pending queue is non-empty, any path leading to an infinite kernel wait is wrong.

Root Cause: sd_event_wait() Return Value Semantics Incorrectly Overridden

Old Logic Assumption: wait(0) Would Not Return 0 When Pending Exists

The main loop of sd_event_run() simplifies to:

r = sd_event_prepare(e);
if (r == 0)
    r = sd_event_wait(e, timeout);
if (r > 0)
    r = sd_event_dispatch(e);

Return values control flow: prepare() returns positive if events are ready (go to dispatch), zero if none (go to wait). wait() returns positive if kernel collected events, zero if no events or internal state requires restart. When prepare() found pending events, it still called sd_event_wait(e, 0) to collect any kernel-ready events at that moment. The old code assumed that with pending already present, wait(0) would never return 0. If it did, the code simply reset state to ARMED and returned 0.

v250 Introduced New Path: Ratelimit Expiry Callback Makes wait(0) Return 0

Commit fd69f2247520 (systemd v250) added a ratelimit expiry callback. When an event source exceeds its burst limit, it is temporarily disabled and re-enabled via a timer. At timer expiry, process_timer() runs the callback, which may re-enable the source or modify its timer. Because the callback can change state, sd_event_wait() returns 0 and sets state to INITIAL, forcing the caller to re-run prepare(). This return-0 means "state changed, recalculate", not "no work".

The bug emerges when these two paths combine:

prepare() finds user-space pending
↓
wait(0) reads ratelimit timerfd
↓
expiry callback runs, state goes to INITIAL, returns 0
↓
prepare() overwrites state to ARMED, returns 0
↓
sd_event_run thinks no pending, enters wait(UINT64_MAX)
↓
epoll_wait(-1) blocks forever

In the production case, mount_on_ratelimit_expire() re-enabled the manager run queue's defer source (a pure user-space source that does not generate epoll readiness). The ratelimit timerfd had just been consumed, so the kernel ready list was empty. The old code skipped the next prepare(), leaving nothing to wake PID 1. Version bisection confirmed the defect from commit fd69f2247520; commit edc027b4f1cf added the mount expiry callback, enabling the trigger in PID 1. Affected versions included TencentOS Server 4's systemd 255, upstream v261.3, v262, and main at the time.

Minimal PoC: Stable Reproduction Using Pipe and Defer Source

To turn the low-probability timing into a deterministic test, the PoC preserves three conditions: (1) an event source enters ratelimit and its expiry callback fires; (2) the callback consumes the last kernel readiness; (3) user-space pending remains but cannot wake epoll. The PoC uses a non-blocking pipe to simulate a mount watch source and a defer source to simulate the manager run queue. The pipe is kept readable to trigger ratelimit quickly; the expiry callback drains the pipe. At that point the timerfd is consumed, the pipe is no longer readable, the kernel ready list is empty, and only the defer source stays in the user-space pending queue. Affected versions reliably enter epoll_wait(-1).

A control mode ( --no-drain) leaves the pipe readable; even if the loop incorrectly enters epoll, the pipe readiness wakes it, and the test passes. Results across versions:

TencentOS 4 (systemd 255): 20/20 hangs in trigger mode, 20/20 passes in control.

Upstream v261.3, v262, main: stable hang in trigger, stable pass in control.

Fixed libsystemd: 20/20 passes in both modes. strace confirms the sequence:

epoll_pwait2(..., timeout=0) = 1   # prepare's non-blocking wait
read(timerfd, ...) = 8             # consume ratelimit expiry
read(pipe, "x", ...) = 1           # expiry callback drains pipe
read(pipe, ...) = -1 EAGAIN
 epoll_wait(..., -1)                # no new kernel events, waits forever

Fix: Explicit State Flag to Restore Correct Event Loop

The first fix attempt added a goto retry in sd_event_prepare() when wait(0) returns 0, re-running prepare from the start. This cleared the hang but risked running multiple ratelimit expiry callbacks in one public API call, breaking the "one callback per call" scheduling guarantee.

The final fix introduces a ratelimit_expired flag in the sd_event object, making "expiry callback executed this round" an explicit state:

if (e->ratelimit_expired) {
    e->state = SD_EVENT_PENDING;
    return 1;
}

sd_event_wait()

now returns PENDING (positive) instead of 0 after an expiry callback. sd_event_dispatch() detects this flag, transitions state to INITIAL, and returns 1 without dispatching another callback. The caller then naturally proceeds to the next prepare(), where timerfd is re-armed and pending re-evaluated, preserving the single-callback-per-round boundary.

A subtle detail: process_timer() must finish processing all expired timer queues before returning, because multiple timerfds may have been read by process_epoll(). Returning early would leave some timers unprocessed and unarmed. The fix ensures all timer queues are handled before checking the flag.

Regression tests assert that sd_event_prepare() returns positive and that dispatch() does not execute a second callback in the same round. The fix was backported to TencentOS Server 4's systemd 255 (which uses CLOCK_MONOTONIC for these timers) by applying the same state migration to the monotonic timer path. Post-fix, both trigger and control modes pass 20/20 runs.

Conclusion

The real failure was not the mount process but the broken contract between prepare() 's return value and the event loop state. The old code misinterpreted "state changed, re-prepare" as "no events", causing an event loop with pending work to sleep indefinitely. The fix makes the ratelimit expiry an explicit state, letting wait(), dispatch(), and the next prepare() each do their proper job. User-space pending is no longer ignored, and the callback dispatch boundary remains unchanged.

The fix has been merged upstream (systemd PR #44108, https://github.com/systemd/systemd/pull/44108, commit https://github.com/systemd/systemd/commit/1a60b10de9e4a3fe59a2efa3c6a58fd72c305807) and backported to TencentOS Server 4.