Why Did My Spring Cloud Gateway Crash? Uncovering Inode Exhaustion and the Fix

Timely Handling

The gateway (Spring Cloud Gateway) returned 500 errors, so the author simply restarted the service, which appeared to resolve the issue temporarily.

Problem Investigation

Although no new deployment had been made, a massive influx of requests (estimated 3 million over two days) caused the gateway to fail. Logs showed an error when creating a temporary directory, despite ample disk space.

The failure originated from the Spring class SynchronossPartHttpMessageReader, which creates a temporary directory for each request to store multipart data, even when the request is not multipart.

Each request therefore generated a new temporary directory on the server.

Because the number of requests was huge, the OS ran out of inode entries, causing every subsequent request to fail with a 500 error.

inode

In Unix‑like file systems, an inode stores a file’s metadata (size, timestamps, permissions, etc.). The number of inodes is limited; when they are exhausted, new files or directories cannot be created even if there is free disk space.

tmp

The /tmp directory is meant for temporary files and is usually cleaned by a daemon such as tmpwatch. Restarting the system also clears its contents.

Solution

Use tmpwatch to clean old temporary files more aggressively.

Avoid using HttpMessageReader for non‑multipart requests.

Schedule periodic restarts.

Modify the source code to create temporary directories only when parsing multipart data.

Wait for an upstream fix in a newer version.

The author tried the first three options but found them unsatisfactory. Instead, they searched the Spring Cloud Gateway GitHub repository and found related issues where maintainers discussed the behavior.

Maintainers explained that the temporary directory is created to avoid security risks of a fixed location, and that the implementation was not intended as a bug. However, they later acknowledged that massive directory creation could exhaust inodes and adjusted the code so that directories are only created for multipart parsing.

The fix was released in Spring Cloud Gateway version 5.2.16. The new implementation creates a single random temporary directory and reuses it, preventing inode exhaustion.

After upgrading the gateway to version 5.2.16, the 500 errors disappeared and the service remained stable.