Why Does Linux Report “Out of Memory” When PID Allocation Fails?

A server repeatedly failed to execute any command, showing the error "fork: cannot allocate memory" despite abundant free RAM. The root cause turned out to be a handful of Java processes spawning too many threads, exhausting the PID space rather than actual memory.

1. Underlying Mechanism

Linux creates both processes and threads via the core do_fork function, which copies the current task’s data structures. The key steps are:

//file:kernel/fork.c
long do_fork(unsigned long clone_flags, ...)
{
    // copy_process performs the heavy lifting
    p = copy_process(clone_flags, stack_start, stack_size,
                     child_tidptr, NULL, trace);
    ...
}

The copy_process routine allocates a new task_struct and invokes a series of copy_* helpers, including alloc_pid to obtain a PID.

//file:kernel/fork.c
static struct task_struct *copy_process(unsigned long clone_flags, ...)
{
    ...
    // Allocate integer PID value
    if (pid != &init_struct_pid) {
        retval = -ENOMEM;
        pid = alloc_pid(p->nsproxy->pid_ns);
        if (!pid)
            goto bad_fork_cleanup_io;
    }
    p->pid = pid_nr(pid);
    p->tgid = p->pid;
    if (clone_flags & CLONE_THREAD)
        p->tgid = current->tgid;
    ...
bad_fork_cleanup_io:
    if (p->io_context)
        exit_io_context(p);
    ...
    return ERR_PTR(retval);
}

If alloc_pid fails, the kernel unconditionally sets retval = -ENOMEM, causing the user‑space message "cannot allocate memory" even when the failure is unrelated to actual memory.

1.1 How alloc_pid Works

//file:kernel/pid.c
struct pid *alloc_pid(struct pid_namespace *ns)
{
    // 1) Allocate the pid object itself
    pid = kmem_cache_alloc(ns->pid_cachep, GFP_KERNEL);
    if (!pid)
        goto out;
    // 2) Allocate an integer PID number via alloc_pidmap
    for (i = ns->level; i >= 0; i--) {
        nr = alloc_pidmap(tmp);
        if (nr < 0)
            goto out_free;
        pid->numbers[i].nr = nr;
        pid->numbers[i].ns = tmp;
        tmp = tmp->parent;
    }
    ...
out:
    return pid;
out_free:
    goto out;
}

The kernel represents a PID as a small struct pid object, not a simple integer. Allocating this object can fail due to memory shortage, but allocating the numeric PID can also fail (e.g., when the PID namespace is exhausted). In both cases the kernel still returns -ENOMEM, which is misleading.

In container environments a single process may need multiple PID numbers—one in the container namespace and one in the host namespace—so the loop may allocate several IDs.

2. Does a Newer Kernel Fix It?

The author examined Linux 5.16.11, where copy_process now propagates the actual error from alloc_pid instead of hard‑coding -ENOMEM:

//file:kernel/fork.c
pid = alloc_pid(p->nsproxy->pid_ns_for_children, args->set_tid, args->set_tid_size);
if (IS_ERR(pid)) {
    retval = PTR_ERR(pid);
    goto bad_fork_cleanup_thread;
}

However, the alloc_pid implementation still defaults to -ENOMEM for most failure paths, with a comment explaining that the error code cannot be changed without breaking user‑space expectations.

3. Takeaways

The “fork: cannot allocate memory” message can be triggered by PID exhaustion, not actual RAM shortage.

When encountering this error, first check the number of processes/threads (e.g., via ps -eLf | wc -l) before assuming a memory problem.

Increasing /proc/sys/kernel/pid_max can raise the PID limit, but the fundamental fix is to identify and stop the runaway processes or threads.

Even recent kernels have not fully resolved the misleading ENOMEM mapping for PID allocation failures.