Troubleshooting OceanBase Backup Clean Stuck Issue and NFS Configuration Recommendations

1 Problem Background

The OceanBase cluster is configured to perform a full backup daily and retain backup files for four days, but up to 13 backup files were observed, causing high disk usage.

2 Investigation

Normally a cleanup task runs periodically; the excess files suggested the cleanup task was failing.

2.1 Cleanup Task Details

The automatic cleanup interval is calculated as min(recovery_window/2, 1d) , so a cleanup should run once per day. For versions prior to oceanbase-3.1.2-20211230114204 , the interval is recovery_window/2 .

The cleanup task was found to be stuck:

MySQL [oceanbase]> select * from __all_tenant_backup_clean_info;
+----------------------------+----------------------------+-----------+--------+----------------------------+----------------------------+-------------+------------------------+--------+------------------+-----------+---------+------------------+--------+---------+
| gmt_create                 | gmt_modified               | tenant_id | job_id | start_time                 | end_time                   | incarnation | type                   | status | parameter        | error_msg | comment | clog_gc_snapshot | result | copy_id |
+----------------------------+----------------------------+-----------+--------+----------------------------+----------------------------+-------------+------------------------+--------+------------------+-----------+---------+------------------+--------+---------+
| 2021-03-19 14:18:50.841475| 2024-10-19 00:28:50.021733| 1         | 1187   | 2024-10-19 00:28:49.998736| 2024-10-19 00:27:09.663132| 1           | DELETE OBSOLETE BACKUP| DOING  | 1728923329993025 |           |         | 1728845993342237 | 0      | 0       |
+----------------------------+----------------------------+-----------+--------+----------------------------+----------------------------+-------------+------------------------+--------+------------------+-----------+---------+------------------+--------+---------+
2 rows in set (0.01 sec)

The view __all_rootservice_event_history showed related events, confirming the cleanup task was abnormal.

2.2 Log Analysis – Scheduler Failure

[root@hwc-ocp RS_cl_bigdata]# grep "ob_backup_data_clean" rootservice.log.20241022040332 | grep WARN
... schedule_sys_tenant_backup_data_clean ... sys tenant clean info status is not stop, can do scheduler(ret=-9044, ...)
... failed to schedule sys tenant backup(ret=-9044)

The keyword "sys tenant clean info status is not stop, can do scheduler" indicated the system‑tenant cleanup status was not STOP , causing the scheduler to fail.

2.3 Log Analysis – Path Recognition Failure

... failed get_file_range(uri=file:///obbak_bigdata/.../index/1102810163026406/13, ret=-4018 ...)
... failed get_index_file_id_range ... ret=-4018

The error code -4018 showed the backup clean task could not recognize the file path.

2.4 Log Analysis – I/O Call Exceptions

... access storage op=list_files uri=file:///obbak_bigdata/... speed=0.00 MB/s
... access storage op=storage reader ... speed=0.00 MB/s

Zero‑speed I/O operations suggested an NFS‑side problem.

2.5 NFS Check

[admin@x-shhp-oceanbase-db-p02:~]$ mount | grep nfs
2.4.8.124:/obbak_bigdata on /obbak_bigdata type nfs (rw,vers=3, ...)

The production environment used NFS version 3, which is a non‑standard deployment for OceanBase and can lead to the observed anomalies.

3 Optimization Suggestions

Configure NFS strictly according to the official documentation, upgrade the NFS version to 4.x, and the backup cleanup blockage is resolved.

References

[1] OceanBase NFS Deployment Recommendations: https://www.oceanbase.com/docs/enterprise-oceanbase-database-cn-10000000000360550