Turning CDB OSS Failures into Wins: A Practical Backup & Monitoring Guide

Background

During a handover the primary CDB OSS server suffered a hard‑disk failure, rendering its configuration file unreadable. Because the location of the standby OSS was stored only in that file, the team could not identify the backup machine.

Problem Analysis

The incident revealed three gaps:

No independent backup of OSS configuration; some OSS instances lacked a standby altogether.

Standby OSS machines were not verified for operability.

OSS database primary‑secondary replication was not monitored, and several OSS DBs had no secondary.

Operational Response

An eight‑step remediation plan was implemented to achieve full disaster‑recovery coverage and automated verification.

Inventory all CDB clusters to record OSS disaster‑recovery status: presence of a standby, standby health, hardware model, warranty status.

Inventory OSS DB disaster‑recovery status: presence of a secondary, synchronization health.

Develop and schedule scripts that back up OSS data for every CDB cluster (e.g., tar/compress the OSS data directory and store it in a secure backup repository).

Develop monitoring scripts that query OSS DB replication lag (using

SELECT MASTER_LOG_FILE, MASTER_LOG_POS FROM performance_schema.replication_connection_status

or equivalent) and raise alerts if lag exceeds a threshold.

Develop scripts to back up the primary data of each OSS DB (e.g., logical dump with mysqldump or physical backup with xtrabackup).

Document the procedures for deploying a new OSS instance and adding a secondary OSS, updating existing installation guides.

Perform a controlled failover of each OSS cluster to its secondary, record the exact commands and required configuration changes, and produce step‑by‑step operational documentation.

After backing up primary OSS and DB, execute restoration tests (restore the OSS data archive and reload the DB dump) to verify backup integrity and recovery time.

Key Practices

Automate inventory collection to maintain an up‑to‑date view of disaster‑recovery assets.

Store backups in a versioned, off‑site location and verify them regularly.

Implement continuous replication health monitoring with alerting.

Maintain reproducible failover runbooks to reduce mean time to recovery.