Why Bash Memory Allocation Fails After Update: 32‑bit vs 64‑bit Process Layout Explained

Problem Overview

After updating Bash on a specific class of servers, SSH logins occasionally failed with an error indicating that Bash could not allocate memory. The error appeared sporadically, suggesting a memory‑allocation issue during Bash startup.

Initial Investigation

Tracing the failure with strace on the sshd process showed that Bash itself allocated only a few bytes before exiting, indicating that Bash’s own memory usage was not the cause. However, Bash relied solely on brk() for memory management, without using malloc or mmap, which is unusual but not inherently problematic.

Process Memory Layout

On Linux, a process’s virtual address space is typically divided (from high to low addresses) into kernel‑space, user stack, mmap region, brk region, data segment, and code segment. The exact boundaries depend on system settings such as randomize_va_space. The article includes diagrams (Fig‑0 – Fig‑5) illustrating the expected layout versus the observed abnormal layout.

The abnormal layout showed the brk region compressed to a tiny range (0x886000‑0x7ac000) because a dynamic library was mapped directly after the data segment, leaving insufficient space for further brk expansions.

32‑bit vs 64‑bit Differences

When the same Bash binary was started manually (64‑bit) the layout was normal, but the Bash spawned by sshd (which was 32‑bit) exhibited the compressed layout. A small test program compiled as 32‑bit reproduced the problem, while the 64‑bit build behaved correctly. This indicated that the issue was tied to the 32‑bit execution path.

Kernel Exec‑Shield Interaction

CentOS 6.5 uses the Red Hat “exec‑shield” security feature, which modifies how the kernel searches for executable memory regions. The kernel parameter /proc/sys/kernel/exec-shield can be set to 0‑3, controlling whether the exec‑shield logic is applied.

The function arch_pick_mmap_layout() selects the mmap layout based on exec‑shield settings. For 32‑bit processes that request executable memory, the kernel may start the search at mm->shlib_base (derived from SHLIB_BASE plus a random offset). Because the ELF loader uses mmap with MAP_FIXED for subsequent segments, the data segment can be forced into a low address range, squeezing the brk region.

Transition from 32‑bit to 64‑bit Process

The ELF loader function load_elf_binary() is shared by both 32‑bit and 64‑bit binaries. When a 32‑bit process execs a 64‑bit binary, the kernel first marks the task as 32‑bit, performs flush_old_exec(), and only later clears the 32‑bit flag with SET_PERSONALITY(). However, the mm->get_unmapped_exec_area pointer, set earlier for the 32‑bit layout, is not cleared, so the newly created 64‑bit process still uses the exec‑shield‑driven search range, causing the same compressed brk region.

Work‑arounds

Set ulimit -s unlimited and disable exec‑shield (set /proc/sys/kernel/exec-shield to 0 or 1) before launching the process, forcing the kernel to use the traditional brk allocation path.

Disable randomize_va_space, though this merely hides the symptom.

Both approaches mitigate the symptom but do not address the underlying kernel bug. The safest practical recommendation is to avoid using a 32‑bit parent process to start 64‑bit binaries on affected systems.

Conclusion

The intermittent Bash login failures were caused by a kernel‑level interaction between exec‑shield’s 32‑bit executable‑memory allocation strategy and the transition from a 32‑bit sshd process to a 64‑bit Bash. Adjusting exec‑shield settings or using a 64‑bit parent process resolves the issue.