Why Did My Server’s Load Spike to 360? Uncovering a Hidden Postdrop Crisis

Yesterday a development server became inaccessible: SSH login was extremely slow, commands barely responded, and the website was down. Running top showed a terrifying load average of 360 and an enormous number of tasks, indicating the system was practically paralyzed.

To restore operability, the load was first reduced. The top output revealed many postdrop processes, suggesting they were the culprit. They were stopped to give the system a chance to breathe:

# ps -ef | grep postdrop | grep -v grep | awk '{print "kill -9 " $2}' | sh

Log inspection in /var/log/maillog showed repeated warnings:

Apr 21 16:56:13 AY140 postfix /postdrop[2377]: warning: mail_queue_enter: create file maildrop/157768.2377: No space left on device

The disk usage was 94% ( # df -h), but the inode usage was 100% ( # df -i), meaning no inodes were available. The urgent task was to free inode space.

The offending directory was identified as /var/spool/postfix/maildrop . Its size was over 4 GB: # du -sh . All files inside were removed: # ls | xargs rm -f After the cleanup, disk usage dropped dramatically and the system became responsive again.

Further investigation showed that a cron job was launching sendmail , which in turn started the numerous postdrop processes. The cron task was generating error output that triggered email notifications.

The root cause was a failing program run by the cron job. The simple fix was to suppress the cron job’s output by appending &>/dev/null to the command, preventing error messages from being mailed.

After applying this change, top showed no more postdrop processes, the load average returned to normal, and the server operated smoothly.