Understanding the Misleading ENOMEM Error in Linux Process Creation

When a user encounters the error "fork: cannot allocate memory" on a Linux server, the immediate suspicion is insufficient memory, yet memory usage may be normal; the true cause can be an excessive number of threads or PID allocation problems.

The investigation begins by examining the kernel's process creation path, which centers on the do_fork function. This function ultimately calls copy_process , which creates a new task_struct by copying various kernel data structures.

Key excerpts from the kernel source illustrate this flow:

//file:kernel/fork.c long do_fork(unsigned long clone_flags, ...) { ... p = copy_process(clone_flags, stack_start, stack_size, child_tidptr, NULL, trace); ... }

The copy_process routine allocates a PID via alloc_pid . If alloc_pid fails, the code pre‑sets retval = -ENOMEM and returns that error, regardless of whether the failure was due to memory exhaustion or inability to allocate a PID number.

//file:kernel/fork.c static struct task_struct *copy_process(...){ ... if (pid != &init_struct_pid) { retval = -ENOMEM; pid = alloc_pid(p->nsproxy->pid_ns); if (!pid) goto bad_fork_cleanup_io; } ... p->pid = pid_nr(pid); p->tgid = p->pid; if (clone_flags & CLONE_THREAD) p->tgid = current->tgid; ... }

The alloc_pid function first tries to allocate a struct pid object, then obtains a numeric PID via alloc_pidmap . Both failure paths ultimately return -ENOMEM , even when the underlying reason is not memory shortage.

//file:kernel/pid.c struct pid *alloc_pid(struct pid_namespace *ns) { pid = kmem_cache_alloc(ns->pid_cachep, GFP_KERNEL); if (!pid) goto out; // allocate numeric PID for (i = ns->level; i >= 0; i--) { nr = alloc_pidmap(tmp); if (nr < 0) goto out_free; pid->numbers[i].nr = nr; pid->numbers[i].ns = tmp; tmp = tmp->parent; } ... return pid; out_free: ... return NULL; }

Because the kernel historically exposed ENOMEM to user space for PID allocation failures, users are misled into thinking the system is out of memory. Newer kernels (e.g., 5.16) still retain this behavior, as indicated by comments in the source.

In conclusion, when encountering ENOMEM during fork, one should first check the number of processes/threads and PID limits ( /proc/sys/kernel/pid_max ) rather than assuming a memory shortage.