Why Did My Oracle Archive Log Fill Up? A Step‑by‑Step RAC Backup Failure Diagnosis

Incident Overview

A client reported that the Oracle archive log directory was full, preventing business operations. After logging in, the DBA confirmed the BACK disk group was indeed full, causing log writes to fail.

Immediate Recovery

The DBA used asmcmd to navigate to the archive directory and manually removed the oldest archive logs (e.g., those from the 15th day). This quick deletion restored normal business functionality.

Root Cause Investigation

Further investigation revealed that the environment had a weekly backup script scheduled via crontab: incremental backups Monday‑Saturday and a full backup on Sunday. The script also cleans up old archive logs after a successful backup.

Log analysis showed that after the 15th day no backups had been executed. Checking the backup logs displayed an ORA-01017 error indicating an invalid username/password.

Reviewing the backup script’s login statements uncovered that the SYS password had been changed on only one RAC node. Consequently, when the SCAN IP routed connections to node 2, the SYS login failed, while connections to node 1 succeeded.

Resolution

The DBA synchronized the SYS password across both RAC nodes. Afterward, the backup script was run manually and completed successfully, automatically cleaning up the expired archive logs.

Fault Analysis

Changing the SYS password on a single node caused the backup script to fail due to ORA-01017 errors, preventing the post‑backup archive cleanup and leading to the archive directory overflow.

Recommendations

When modifying the SYS password in a RAC environment, update the password on all nodes simultaneously.

Deploy a monitoring script that checks archive directory usage and forces cleanup of old archives when a defined threshold is exceeded.

Illustrative Screenshots