Recovering PostgreSQL After Power Loss: Fixing Invalid Checkpoint Errors

Problem

Following a power outage, the PostgreSQL database failed to start, returning the error psql: could not connect to server: No such file or directory. The error screenshot is shown below.

Background

The database runs as a single‑node PostgreSQL pod on Kubernetes, with its data directory mounted from a host path. Two Kubernetes clusters in the same data center had overlapping pod CIDR ranges, causing occasional IP conflicts. After correcting the pod network and restarting all pods, the PostgreSQL pod could not start, and container logs displayed the above error.

Solution Steps

1. Verify socket file existence

Enter the container and check for the Unix socket file: kubectl exec -it -n namespace <em>containerId</em> /bin/sh Inside the container, /var/run/postgresql/.s.PGSQL.5432 was missing, indicating the PostgreSQL process was not running.

2. Inspect PostgreSQL process PID

Read the master PID file: cat /var/lib/postgresql/11/main/master.pid The file contains five fields:

154 – main process ID (matches ps -ef | grep postgres)

/var/lib/postgresql/11/main – data directory

1724468744 – start timestamp (needs conversion)

5432 – listening port

/var/run/postgresql – Unix socket directory

Running top confirmed that no PostgreSQL process was active.

3. Attempt to start PostgreSQL

Execute the startup command:

./usr/lib/postgresql/11/bin/pg_ctl -D /var/lib/postgresql/11/main start

The output contained “invalid primary checkpoint record”, meaning the checkpoint data was corrupted.

4. Repair the checkpoint

Use the built‑in tool pg_resetwal to fix the checkpoint:

./usr/lib/postgresql/11/bin/pg_resetwal –D /var/lib/postgresql/11/main

5. Restart the database

After the repair, start the service again:

./usr/lib/postgresql/11/bin/pg_ctl -D /var/lib/postgresql/11/main start

Connection via Navicat succeeded, confirming the database was operational.

Conclusion

The root cause was data file corruption caused by an abnormal restart after a power loss, which broke the primary checkpoint. Repairing the checkpoint with pg_resetwal and restarting the service restored normal operation.