When the Green Light Lies: 50 M Syscalls Cause C++ to Sneakily Consume 4 GB of Memory

Background and Test Setup

We built a syscall‑intensive service that loops roughly 50 million times, performing only raw read operations. Two implementations were prepared: one in C++ and one in Rust, with identical design, workload, and expectations. The goal was to observe any performance or resource differences under extreme pressure.

Initial Observations

Both versions showed nearly identical CPU usage, latency graphs, and throughput, giving a false sense of stability. No crashes, alerts, or error logs appeared, and the system reported everything as normal.

Memory Drift in the C++ Version

After the test began, the C++ process’s memory usage started to climb slowly—not in sudden jumps but as a steady, seemingly harmless increase. Over time the consumption exceeded 4 GB, eventually causing the node to be terminated by the operating system. The Rust version’s memory remained flat throughout.

Code Review of the C++ Implementation

void handle(int fd){
    std::string data;
    char buf[1024];
    while (read(fd, buf, 1024) > 0) {
        data += std::string(buf);
    }
    cache.push_back(data); // grows forever
}

The snippet passed code review because it uses no raw pointers and appears straightforward under low load. However, under high pressure the cache vector grows without any eviction policy, leading to gradual memory bloat.

Why Subtle Failures Are More Dangerous

A classic crash produces obvious symptoms that force immediate attention. In this case, requests continued to succeed, logs stayed clean, and metrics never crossed configured thresholds. The system failed precisely because it *looked* healthy, making the problem harder to detect.

Rust Implementation Remains Stable

fn handle(fd: RawFd) {
    let mut data = Vec::new();
    let mut buf = [0u8; 1024];
    while read(fd, &mut buf) > 0 {
        data.extend_from_slice(&buf);
    }
    process(&data);
}

Rust follows the same logic but does not retain data after the function returns, preventing any hidden long‑term growth.

Production‑Level Architecture Impact

+-----------+
|  Client   |
+-----+-----+
      |
      v
+-----+-----+   +-----+-----+
| Listener  |   | Worker 1 |
+-----+-----+   +-----+-----+
      |               |
      +-------+-------+
              |
              v
          +-----+-----+
          |  Kernel  |
          +-----------+

Each worker processes requests in a loop, adding a small amount of memory pressure per iteration. Individually insignificant, these increments accumulate across many workers, eventually causing a slow‑burn crash.

Consequences During Recovery

When a failing node restarts, traffic shifts to remaining nodes, accelerating their memory consumption and exposing the degradation pattern. The system does not fail from overload but from the cumulative effect of unnoticed memory growth.

Language‑Level Takeaways

C++ offers fine‑grained control, which can yield high performance but requires rigorous discipline to avoid hidden leaks. Rust shifts responsibility to the compiler and runtime, eliminating entire classes of memory‑related bugs before deployment, though it is not a panacea.

Final Reflections

The episode reshaped our understanding of system reliability: stability, not raw speed, is the true metric of value. Gradual, undetected memory bloat can masquerade as normal operation, leading to severe production incidents.