Resolving Missing Core Dumps When Using Supervisor on CentOS 6.5

Background: C/C++ applications often crash, and to reduce manual intervention they are placed under supervisord . While Supervisor prevents processes from staying down, it also disables core dump generation because the core file size limit is set to zero in the supervised environment.

Investigation: After confirming that the core_pattern is unchanged and ulimit -c is unlimited in a normal shell, the author reproduced the crash manually and observed that the supervised process had ulimit -c 0 . Comparing environment variables revealed that Supervisor uses Python's resource module, which defaults RLIMIT_CORE to 0.

Code examples used for testing:

#sysctl -a | grep core_pattern

#include
int main(){
    char *str="this is a Segmentation fault";
    str[0]='f';
    return 0;
}

#!/usr/bin/env python
#coding=utf8
from time import sleep
import os, signal

def child_handler(signum, stackframe):
    while 1:
        try:
            result = os.waitpid(-1, os.WNOHANG)
        except:
            break
        print "Reaped child process %d" % result[0]

signal.signal(signal.SIGCHLD, child_handler)

try:
    pid = os.fork()
    if pid == 0:
        os.execv("/tmp/a.out", [])
except OSError, e:
    pass

sleep(30)

Root cause: Supervisor's set_rlimits only sets RLIMIT_NOFILE and RLIMIT_NPROC, leaving RLIMIT_CORE at the default (0). Consequently, supervised processes cannot write core files.

Solutions:

Temporary work‑arounds without restarting the process: use gdb to change limits (kernel >2.6), echo values into /proc/ /limits , or use prlimit (kernel >2.6.36).

Permanent fix: edit /usr/lib/python2.6/site-packages/supervisor/options.py (or the appropriate path) and add after line 1293: soft, hard = resource.getrlimit(resource.RLIMIT_CORE) resource.setrlimit(resource.RLIMIT_CORE, (-1, hard)) This sets the core file size limit to unlimited for all supervised programs.

After applying the patch, core dumps are generated correctly, allowing developers to locate the exact faulting instruction.

The analysis ends here.