Why Does Docker’s runc Exec Fail? A Deep Dive into Resource Limits and Troubleshooting

Problem Description

During a routine on‑call investigation the system log repeatedly showed Docker error messages such as "stream copy error: reading from a closed fifo" and an OCI runtime exec failure with the message "read init‑p: connection reset by peer".

Investigation Steps

The investigation follows two main steps: (1) identify the component that caused the failure, and (2) determine why that component failed.

Using docker ps, docker‑containerd‑ctr exec, and docker‑runc exec the author isolates the problem to runc , which returns the error.

The runc error log indicates the cause is Resource temporarily unavailable, a typical symptom of hitting OS limits such as thread count, file descriptors, or memory.

Monitoring the affected container shows that the thread count has reached the default Kubernetes limit of 10,000 threads, which explains the failure.

runc Workflow Overview

To clarify how runc works, the article walks through an runc exec example:

runc exec starts a child process runc init. runc init sets up the container namespaces using C constructor tricks and a double clone to create parent, child, and grandchild processes.

The grandchild remains in the new namespace, receives configuration from the parent via a socket pair, and finally calls system.Execv to run the user command.

runc exec adds the grandchild to the container’s cgroup and writes the exec configuration to the pipe.

Steps 2.c and 3 run concurrently, and communication between runc exec and runc init relies on a socket pair (init‑p and init‑c).

Key Code Snippet

func (p *setnsProcess) start() (err error) {
    defer p.parentPipe.Close()
    err = p.cmd.Start()
    p.childPipe.Close()
    if err != nil {
        return newSystemErrorWithCause(err, "starting setns process")
    }
    if p.bootstrapData != nil {
        if _, err := io.Copy(p.parentPipe, p.bootstrapData); err != nil {
            return newSystemErrorWithCause(err, "copying bootstrap data to pipe")
        }
    }
    if err = p.execSetns(); err != nil {
        return newSystemErrorWithCause(err, "executing setns process")
    }
    if len(p.cgroupPaths) > 0 {
        if err := cgroups.EnterPid(p.cgroupPaths, p.pid()); err != nil {
            return newSystemErrorWithCausef(err, "adding pid %d to cgroups", p.pid())
        }
    }
    if err := utils.WriteJSON(p.parentPipe, p.config); err != nil {
        return newSystemErrorWithCause(err, "writing config to pipe")
    }
    ierr := parseSync(p.parentPipe, func(sync *syncT) error {
        switch sync.Type {
        case procReady:
            panic("unexpected procReady in setns")
        case procHooks:
            panic("unexpected procHooks in setns")
        default:
            return newSystemError(fmt.Errorf("invalid JSON payload from child"))
        }
    })
    if ierr != nil {
        p.wait()
        return ierr
    }
    return nil
}

Conclusion

The root cause of the exec failure is a resource limit (thread count) reached inside the container, which runc reports as "Resource temporarily unavailable". Adjusting the cgroup limits or reducing thread usage resolves the issue.

Reference

https://www.kernel.org/doc/Documentation/cgroup-v1/pids.txt

https://github.com/opencontainers/runc