Why Your systemd Service Hits 'Too Many Open Files' Despite limits.conf
This article explains the four layers of Linux file descriptor limits, why /etc/security/limits.conf doesn't affect systemd services, how to configure LimitNOFILE and TasksMax correctly, and provides troubleshooting commands, PromQL queries, and real-world failure case studies for production environments.
Four Layers of File Descriptor Limits
The "too many open files" error (errno 24) can originate from four distinct layers:
Kernel per-process ceiling : /proc/sys/fs/nr_open (default 524288). setrlimit exceeding this returns EPERM even for root.
Process RLIMIT_NOFILE : soft/hard limits. Hard must be ≤ nr_open; exceeding soft returns EMFILE.
System-wide fd total : /proc/sys/fs/file-max (often huge). Exhaustion returns ENFILE affecting all processes.
Application self-limits : internal connection pools; may report business errors instead of errno.
The most common pitfall: LimitNOFILE hard limit cannot exceed nr_open. Setting LimitNOFILE=1048576 on a host with default nr_open=524288 causes immediate service start failure with Failed to set RLIMIT_NOFILE: Operation not permitted.
Two Independent Scopes: PAM vs systemd
/etc/security/limits.confis read only by pam_limits.so during PAM session establishment. systemd services are forked directly by PID 1 and never go through PAM. Therefore limits.conf has zero effect on systemd services.
Three startup paths:
SSH/login : login → pam_limits.so → limits.conf → setrlimit → shell
systemd service : PID 1 → fork → unit's LimitNOFILE= or DefaultLimitNOFILE= → setrlimit → service process
Container : containerd → runc → --ulimit or daemon default; still constrained by host fs.nr_open Practical implications: adding nofile to limits.conf changes ulimit -n in login shells but not for systemd services; su - app / sudo -u app read PAM, systemctl start does not.
systemd Defaults and cgroup v2 Boundaries
systemd's built-in defaults: DefaultLimitNOFILE=1024:524288 (soft=1024, hard=524288) and DefaultTasksMax ≈ 4915 (15% of kernel.pid_max). PID 1 gets higher limits but children fall back to defaults.
rlimit ( getrlimit) controls only 16 resource types (fd, processes, stack, core, locks, msg queues). It cannot limit memory. LimitAS= limits address space (JVM fails), LimitRSS= is no-op on Linux. Real memory limits come from cgroup v2: memory.max / memory.high, pids.max (replaces NPROC), cpu.max, io.max.
Production Baseline Configuration
Example unit app-gateway.service with four categories:
# rlimit four groups
LimitNOFILE=131072
LimitNPROC=32768
LimitCORE=infinity
LimitMEMLOCK=infinity
# cgroup v2 resources
TasksMax=8192
MemoryMax=32G
MemoryHigh=28G
CPUQuota=800%
IOWeight=200Formula for LimitNOFILE: connections × fds_per_connection + internal_fds × 1.5. For Nginx (2 fds/conn) + Java (1.1 fds/conn), 6000 connections → ~18075 → round to 131072 (power of two, aligns with worker_rlimit_nofile).
Coupled parameters that must be set together: LimitNOFILE + fs.nr_open (hard ≤ nr_open), TasksMax + User= + LimitNPROC (UID aggregation), MemoryMax + MemoryHigh (High = 0.85–0.9 × Max), LimitCORE + kernel.core_pattern.
Red lines: never use LimitNOFILE=infinity on bare metal; never set LimitNPROC for root services; never use LimitAS= / LimitRSS= for memory; never set LimitNOFILE=0; daemon-reexec does not apply new limits to running services — must restart.
Verification: Four Layers Must Match
Kernel ceiling: cat /proc/sys/fs/nr_open and file-max systemd's view:
systemctl show app-gateway.service -p LimitNOFILE -p TasksMax -p MemoryMaxProcess actual limits (only trustworthy layer): grep 'open files' /proc/$PID/limits Runtime usage: ls /proc/$PID/fd | wc -l vs soft limit; >50% warrants review, >80% is capacity alert
Use systemctl cat app-gateway.service to see all merged fragments (drop-ins override main unit).
Container Nuances
Container RLIMIT_NOFILE comes from docker run --ulimit, dockerd's unit LimitNOFILE, or runc default. dockerd's LimitNOFILE=infinity propagates to all containers (shows as 1073741816). Containers still constrained by guest kernel nr_open; exceeding it fails with misleading "out of memory" error. Kubernetes pod securityContext has no nofile field; must configure runtime or use initContainer (anti-pattern).
Monitoring Commands and PromQL
Five one-liners cover 90% of scenes: check process limits, fd breakdown by type, system-wide file-nr, systemctl status for tasks/memory, cgroup v2 direct reads ( memory.current, pids.current).
Three PromQL queries (require node_exporter --collector.systemd):
1. Task usage %: systemd_unit_tasks_current / systemd_unit_tasks_max * 100 (>70 warn, >90 critical)
2. Memory pressure:
container_memory_working_set_bytes / container_spec_memory_limit_bytes * 1003. Restart rate: increase(systemd_unit_restart_total[1h]) > 2 indicates too-tight limits.
FD growth rate predicts exhaustion: sample ls /proc/$PID/fd | wc -l every minute; linear growth → hours to exhaustion.
SQL for Fleet-Wide Auditing
Collect per-host per-unit snapshots into host_rlimit table. Key query finds units where soft < 1% of hard but usage >30% of soft — highest risk. Coverage query shows % of units with explicit LimitNOFILE (target as SLO).
cgroup v2 vs v1 Path Differences
v2 unified hierarchy: /sys/fs/cgroup/<unit>/memory.max, pids.max, cpu.max (format "quota period"). stat -fc %T /sys/fs/cgroup returns cgroup2fs. Verify cgroup.subtree_control contains memory else MemoryMax= silently no-ops.
Three Deep-Dive Failure Cases
Case 1: limits.conf Changed, Service Restarted, Soft Still 1024
Payment callback service hitting EMFILE. /proc/$PID/limits showed 1024/524288. Root cause: PAMName= empty (no PAM), unit had no LimitNOFILE=, app didn't self-raise soft. Fix: drop-in with LimitNOFILE=131072, TasksMax=8192. Post-fix: fd usage 0.9%, callback failures 0.
Case 2: LimitNOFILE=1048576 Causes Start Failure
Same service, someone raised limit to 1048576.
systemd[1]: Failed to set RLIMIT_NOFILE: Operation not permitted. Host nr_open=524288 < 1048576. Config copied from node with nr_open=2097152. Boundary test: systemd-run --scope -p LimitNOFILE=524288 /bin/true succeeds, 524289 fails. Fix: either lower unit value (preferred) or raise nr_open via sysctl.d first.
Case 3: unable to create native thread, LimitNPROC Useless
Java service OOMError "unable to create native thread" at 40% heap. ulimit -u 32768, /proc/$PID/limits 32768/32768. systemctl status showed Tasks: 4915 (limit: 4915) — exactly DefaultTasksMax. TasksMax=4915 (default) vs LimitNPROC=32768 (configured). cgroup pids.max=4915 was the real limit. Thread dump: 3841 Tomcat threads ( maxThreads=4000). Fix: reduce maxThreads to 300 (QPS×RT×safety), set TasksMax=1024. Result: threads 268, latency improved, CPU halved.
Audit and Diff Scripts
Script 1 ( rlimit-audit.sh): read-only snapshot of kernel ceilings, cgroup mode, systemd defaults, per-unit effective limits (soft, hard, tasks, TasksMax, used fds), and risk list (soft < 1% hard). Outputs to timestamped file.
Script 2 ( rlimit-diff.sh): capture before/after TSV snapshots, join on unit name, diff soft/hard/tasks/tmax. Usage: capture-before → change → capture-after → diff. Shows exact changes (e.g., soft 1024→131072, hard 524288→131072, tmax 4915→8192).
Risk Matrix and Rollback
Operations ranked by impact/rollback difficulty. DefaultLimitNOFILE requires daemon-reexec (restarts PID 1, management pause) and only affects new units. High-risk change six elements: scope, pre-checks (current value, nr_open headroom, drop-in count), backup ( systemctl cat), low-traffic window, canary with metric gates, one-command rollback (delete drop-in, reload, restart).
Nine Engineering Iron Laws
limits.confdoes not apply to systemd services — use LimitNOFILE=. LimitNOFILE hard ≤ fs.nr_open; exceeding returns EPERM; equality allowed but keep margin. infinity maps to host nr_open; explicit numbers use strict comparison — hence "copy infinity works, copy number fails".
Soft defaults to 1024; app must self-raise (Nginx/MySQL do, most business apps don't).
Limit tasks with TasksMax= (cgroup pids.max), not LimitNPROC= (UID aggregation, no-op for root). LimitAS= and LimitRSS= cannot control memory; use MemoryMax=.
Verify with /proc/PID/limits, not systemctl show. daemon-reexec does not update running services; must restart.
Configuration coverage is an SLO: 78% on default 1024 is "not yet discovered", not "fine".
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
