Understanding Linux fork, alloc_pid and the Misleading ENOMEM Error

Hello everyone, I’m Fei! A reader reported that their online server repeatedly fails any command with the error "fork: cannot allocate memory". The problem appears every 2‑3 days after a reboot.

# service docker stop
-bash fork: cannot allocate memory
# vi 1.txt
-bash fork: cannot allocate memory

Although the message suggests insufficient memory, checking the system shows plenty of free RAM, and the command sometimes succeeds after several attempts.

Fei and the group proposed three possible causes:

1. In a NUMA architecture, the process might be bound to a node, using only that node's memory.

2. If all memory modules are inserted into a single slot, other nodes may appear to have no memory.

3. The number of processes/threads may exceed the system limit.

After investigation, the real cause was the third one: several Java processes on the server created too many threads, triggering the error, not an actual memory shortage.

1. Underlying Process Analysis

The Linux error message is misleading, leading users to look at memory instead of process counts. To understand why, we dive into the kernel.

The reader’s server runs CentOS 7.8 with kernel version 3.10.0‑1127.

1.1 do_fork Dissection

Both process and thread creation eventually call the core function do_fork , which copies the current task to create a new task_struct .

//file:kernel/fork.c
long do_fork(unsigned long clone_flags, ...)
{
    // The creation is essentially a copy of the current process
    // Note: the second‑last argument is NULL
    p = copy_process(clone_flags, stack_start, stack_size,
                    child_tidptr, NULL, trace);
    ...
}

The heavy lifting happens inside copy_process :

//file:kernel/fork.c
static struct task_struct *copy_process(unsigned long clone_flags,
    ...
    struct pid *pid,
    int trace)
{
    // task_struct is the kernel’s representation of a process/thread
    struct task_struct *p;
    ...
    // Duplicate the core data structures
    p = dup_task_struct(current);
    retval = copy_semundo(clone_flags, p);
    retval = copy_files(clone_flags, p);
    retval = copy_fs(clone_flags, p);
    retval = copy_sighand(clone_flags, p);
    retval = copy_mm(clone_flags, p);
    retval = copy_namespaces(clone_flags, p);
    retval = copy_io(clone_flags, p);
    retval = copy_thread(clone_flags, stack_start, stack_size, p);
    // -----
    // Allocate an integer PID value
    if (pid != &init_struct_pid) {
        retval = -ENOMEM;
        pid = alloc_pid(p->nsproxy->pid_ns);
        if (!pid)
            goto bad_fork_cleanup_io;
    }
    // Set the PID in the new task_struct
    p->pid = pid_nr(pid);
    p->tgid = p->pid;
    if (clone_flags & CLONE_THREAD)
        p->tgid = current->tgid;
    ...
bad_fork_cleanup_io:
    if (p->io_context)
        exit_io_context(p);
    ...
fork_out:
    return ERR_PTR(retval);
}

The code shows that regardless of why alloc_pid fails, the kernel sets retval = -ENOMEM , so the user sees an out‑of‑memory error.

In errno-base.h , ENOMEM is defined as:

//file:include/uapi/asm-generic/errno-base.h
#define ENOMEM 12   /* Out of memory */

1.2 Reasons Why alloc_pid Can Fail

Examining alloc_pid reveals two failure paths:

//file:kernel/pid.c
struct pid *alloc_pid(struct pid_namespace *ns)
{
    // 1) Allocation of the pid struct itself fails
    pid = kmem_cache_alloc(ns->pid_cachep, GFP_KERNEL);
    if (!pid)
        goto out;
    // 2) Allocation of an integer PID number fails
    tmp = ns;
    pid->level = ns->level;
    for (i = ns->level; i >= 0; i--) {
        nr = alloc_pidmap(tmp);
        if (nr < 0)
            goto out_free;
        pid->numbers[i].nr = nr;
        pid->numbers[i].ns = tmp;
        tmp = tmp->parent;
    }
    ...
out:
    return pid;
out_free:
    goto out;
}

The kernel represents a PID as a small struct:

//file:include/linux/pid.h
struct pid {
    atomic_t count;
    unsigned int level;
    struct hlist_head tasks[PIDTYPE_MAX];
    struct rcu_head rcu;
    struct upid numbers[1];
};

If the first allocation fails, the error truly is out‑of‑memory. The second case, however, occurs when the kernel cannot obtain a free PID number (e.g., PID namespace limits), yet it still returns ENOMEM, which is misleading.

Note that a process may need to allocate multiple PID numbers—one for its own namespace and one for the parent namespace—especially in containers.

2. Does a Newer Kernel Version Improve This?

The author checked a newer kernel (5.16.11) to see if the error handling changed. Using the online source browser elixir.bootlin.com , the newer copy_process shows:

//file:kernel/fork.c
static __latent_entropy struct task_struct *copy_process(...)
{
    ...
    pid = alloc_pid(p->nsproxy->pid_ns_for_children, args->set_tid,
                    args->set_tid_size);
    if (IS_ERR(pid)) {
        retval = PTR_ERR(pid);
        goto bad_fork_cleanup_thread;
    }
    ...
}

However, the comment inside the newer alloc_pid explains why ENOMEM is still used:

//file:include/pid.c
/*
 * ENOMEM is not the most obvious choice, especially when the child
 * subreaper has already exited and the PID namespace denies creation of
 * new processes. But ENOMEM is what we have exposed to userspace for a long
 * time and it is documented behavior for PID namespaces, so we can't easily
 * change it even if a better error code exists.
 */
retval = -ENOMEM;
...

Thus, even recent kernels retain the misleading ENOMEM return for PID‑allocation failures.

Conclusion

Linux may report "out of memory" when the real problem is PID exhaustion caused by an excessive number of threads or processes. When encountering such errors, check the process/thread count before assuming memory shortage.

Increasing /proc/sys/kernel/pid_max can raise the PID limit, but the fundamental solution is to identify and stop the runaway processes—typically a few tens of thousands of threads is unreasonable for most servers.