Using syslog-ng with daemontools: Design Principles, Operation Modes, Installation, and Troubleshooting

1 syslog-ng

syslog-ng is a replacement for the traditional syslog daemon; it can fully take over syslog services and, through user‑defined rules, provide more powerful filtering capabilities.

(1) Design Principles

One design principle is to achieve finer‑grained message filtering, supporting both content‑based and priority‑based filters. Another principle is to simplify forwarding across different firewall zones by supporting host chains, allowing the original source address to be identified even after multiple hops. The configuration aims to be both powerful and concise.

(2) Working Principle

A message path consists of one or more log sources, one or more filter rules, and one or more log destinations. Messages from a source enter syslog-ng; if a message matches a rule, syslog-ng forwards it to the corresponding destination.

The following process illustrates how client‑side log messages travel to a syslog-ng server.

Steps:

1. A device or application sends log messages to a source on the syslog-ng client, e.g., an Apache web server on Linux writes to /var/log/apache.

2. The syslog-ng client on the web server reads the information from /var/log/apache.

3. The client processes the first log line, which includes the /var/log/apache source.

4. The client optionally performs operations such as filtering, parsing, or rewriting; if the message satisfies all filters, it is sent to the configured destination, for example a remote syslog-ng server.

5. The client processes the next log line, repeating steps 3‑4.

6. The client sends the message to the destination defined on the syslog-ng server.

7. The syslog-ng server reads the message and the first log line that includes the source.

8. The server optionally performs filtering, parsing, or rewriting; if the message passes all filters, it is sent to the configured log destination.

9. The server processes the next log line, repeating steps 7‑9.

(3) Working Modes

The open‑source edition of syslog-ng provides three typical operating modes: client mode, relay mode, and server mode. Detailed documentation can be found at the official syslog-ng guide.

2 daemontools

daemontools is a collection of tools for managing services on UNIX. The supervise program monitors a service, starts it, and automatically restarts it if it stops. Installing supervise is straightforward: you only need a directory containing a script that runs the service. multilog stores error messages and logs, can add timestamps, and can limit disk usage; it does not host daemon threads.

Installation of daemontools:

wget 'http://cr.yp.to/daemontools/daemontools-0.76.tar.gz' && tar zxf daemontools-0.76.tar.gz
cd admin/daemontools-0.76package/install

Note: If installation fails, it may be because daemontools requires the patch daemontools-0.76.errno.patch.

wget 'http://www.qmail.org/moni.csi.hu/pub/glibc-2.3.1/daemontools-0.76.errno.patch' && patch -Np1 -i ../../daemontools-0.76.errno.patch

Alternatively, edit the source file src/error.h, replace extern int errno; with #include <errno.h>.

After a successful installation, verify with:

ps -ef | grep svscan
man svscan

In inittab you will see that daemontools uses the init system: SV:123456:respawn:/command/svscanboot The supervise program runs a service by executing ./run in a directory named s. If ./run exits, supervise restarts it after a one‑second delay to avoid rapid respawn loops.

If a file named down exists in the s directory, supervise will not start ./run immediately. The svc command can be used to start or control the service.

Status information maintained by supervise resides in s/supervise and can be inspected with svstat. If another supervise instance is already running, the new one will exit. Once started, supervise runs until it is terminated or explicitly told to exit; you can check its health with svok and start multiple supervise threads with svscan.

3 daemontools Managing syslog-ng

After installing both daemontools and syslog-ng, create a syslog-ng directory under /service (preferably via a symlink) and add an executable run script with the following content:

#!/bin/bash
exec 2>&1
exec /opt/syslog-ng/sbin/syslog-ng -F --no-caps

Reload the init configuration with telinit q, then use pstree to verify that the service is now supervised:

├─svscanboot─┬─readproctitle
             └─svscan───supervise───syslog-ng

4 Issues Encountered

Initially the run script used exec /opt/syslog-ng/sbin/syslog-ng --no-caps. Observing pstree showed that the system kept spawning new syslog-ng processes.

With assistance, the script was modified to add the -F flag, forcing syslog-ng to run in the foreground, which resolved the problem. The reason is that syslog-ng defaults to background mode; when it daemonizes, the run script exits with status 0, causing supervise to immediately restart it, leading to a loop. Running in foreground allows supervise to monitor the child process via waitpid with the WNOHANG option, preventing rapid respawns.

int wait_nohang(int *wstat) {
#ifdef HASWAITPID
    return waitpid(-1, wstat, WNOHANG);
#else
    return wait3(wstat, WNOHANG, (struct rusage *)0);
#endif
}

void doit(void) {
    ...
    for (;;) {
        r = wait_nohang(&wstat);
        if (!r) break;
        if ((r == -1) && (errno != error_intr)) break;
        if (r == pid) {
            pid = 0;
            pidchange();
            announce();
            if (flagexit) return;
            if (flagwant && flagwantup) trystart();
            break;
        }
    }
    ...
}

References:

http://cr.yp.to/daemontools/supervise.html

http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guides/en/syslog-ng-ose-v3.3-guide-admin-en/html-single/index.html

Article originally from the company wiki “Hotel Development Team’s One Thousand and One Nights”. Qunar colleagues can log in to the wiki for more articles.