Why Did My MongoDB Logs Turn Into PCX Images? Investigation and Fix

Background

MongoDB cluster with four shard nodes, each running on a 16‑core/32 GB CentOS 7.4 server. Log files are stored under

/data

and each day generates about 23‑24 GB of logs.

Incident

A night‑time alert reported high CPU on a shard node. The team assumed the cause was a slow query and attempted to extract the log segment for the high‑CPU period (15:00–16:00), but the log files were identified as “PCX ver. 2.5 image data” instead of the expected ASCII text.

Root Cause Analysis

Log rotation was performed with

echo > log

, which inserts a newline character (0x0a) at the beginning of the file.

The inserted newline matches the header of a PCX image, causing the file type detection to report an image format.

Repeated use of

echo > log

on a locked MongoDB log file overwrote the original content with zeros, further corrupting the file.

Reproduction

Creating an empty file and running

echo > log

shows the file header becomes a newline (0x0a) in hex dump, which is the same header as a PCX image. The same behavior was observed on the production server.

Solution

Replace the manual

echo > log

rotation with the standard MongoDB signal‑based rotation:

<code>kill -SIGUSR1 $(pidof mongod)</code>

A sample init script (

/etc/init.d/mongo_logspit.sh

) is provided to automate the SIGUSR1 rotation and clean logs older than seven days.

Recovery Procedure

Split the corrupted 23 GB log file into 6 GB chunks using

split -b 6G

.

Identify chunks that are still ASCII text and extract the 15:00–16:00 window with

mlogfilter

and

mloginfo

.

Generate slow‑query scatter plots with

mplotqueries

.

Conclusion

Using

echo > log

for log rotation corrupts MongoDB logs by inserting a newline that mimics a PCX header; the proper method is to send

SIGUSR1

to the mongod process. After fixing the rotation, logs return to normal ASCII format and the required time‑range data can be retrieved.