Why Is My Server Overloaded? Diagnosing Nginx FastCGI Temp IO Bottleneck

Problem description A server suddenly triggered a high‑load alarm and the website became slow.

Analysis

(1) Using top showed the CPU iowait exceeding 70%, indicating an IO overload.

(2) iotop -o revealed that Nginx’s write IO was unusually high, and top displayed the Nginx process in state D, meaning it was stuck waiting for IO.

At this point it was clear that Nginx’s heavy write activity caused the load, but the exact files being written were unknown.

(3) The PID of an Nginx worker was obtained and lsof -p <pid> listed open files, showing many entries under fastcgi_temp/xxx besides system libraries and logs.

(4) strace -p <pid> traced the process and confirmed massive writes to a specific file descriptor, matching the files identified by lsof.

(5) iostat 1 displayed heavy write IO on the partition that holds fastcgi_temp, confirming the source of the load.

A hypothesis was that an external client was uploading large files to php‑fpm, but monitoring inbound traffic with EZHTTP showed only modest traffic.

Solution

Since the fastcgi_temp IO pressure could not be eliminated instantly, the temporary fix was to point fastcgi_temp to /dev/shm (memory), restart Nginx, and restore service. A longer‑term fix will require collaboration with developers to address the root cause.

--- Thanks to HaiLin, chief architect of a major e‑commerce site, for providing this case study.