OceanBase Datafile Shrink Procedure and Cluster Maintenance Guide

1 Shrink Scenario

In a bank's 1‑1‑1 OceanBase cluster, an OBServer crash generated a core file in /data/1 whose size (~400 GB) exceeded the remaining free space on the data disk, which had already allocated 90% for block files. This caused an incomplete core file and a full data disk, preventing the node from serving traffic.

After restoring the OBServer service, the team decided to shrink the cluster's data files to 80% of the total disk size to avoid repeat failures.

2 Shrink Operations

Version Information

OBServer version: 3.2.3

OCP version: 3.3.3

Related Parameters

datafile_size

Controls the size of data files. To reduce datafile_size, remove the node from the cluster, rebuild it, and set the parameter (current value is 0).

datafile_disk_percentage

Specifies the percentage of the disk occupied by data_dir. The current cluster value is 90%.

1 Adjust Parameter

Navigate to Cluster → Parameter Management and set datafile_disk_percentage to 80, making the block file occupy 80% of the disk.

2 Reduce Tenant Replicas

Go to Cluster → Tenant Management , select the tenant (e.g., sys), and delete replicas in the desired zone (e.g., zone3) until the task completes.

3 Offline OBServer

In Cluster → Overview , delete the OBServer entry for zone3, which uninstalls the OBServer service on that node.

4 Online OBServer

Since the OceanBase package has been removed from the node, reinstall it and initialize directories before starting the process.

4.1 Install RPM Package

rpm -ivh oceanbase-3.2.3.3-107050022023040817.el7.x86_64.rpm

4.2 Initialize Directories

export cluster_name=sit
mkdir -p /data/1/$cluster_name/{etc3,sort_dir,sstable}
mkdir -p /data/log1/$cluster_name/{clog,etc2,ilog,slog,oob_clog}
mkdir -p /home/admin/oceanbase/store/$cluster_name
chown -R admin:admin /data/1/$cluster_name && chown -R admin:admin /home/admin/oceanbase && chown -R admin:admin /data/log1/$cluster_name
for t in {etc3,sort_dir,sstable}; do ln -sf /data/1/$cluster_name/$t /home/admin/oceanbase/store/$cluster_name/$t; done
for t in {clog,etc2,ilog,slog,oob_clog}; do ln -sf /data/log1/$cluster_name/$t /home/admin/oceanbase/store/$cluster_name/$t; done

4.3 Start OBServer with Parameters

Switch to the admin user, set limits, and launch the observer:

cd /home/admin/oceanbase
ulimit -s 10240   ## stack size
ulimit -c unlimited ## enable core dump
cd /home/admin/oceanbase
/bin/observer -i eth0 -p 2881 -P 2882 -n sit -z zone3 -d /home/admin/oceanbase/store/sit -r '10.186.65.8:2882:2881;10.186.65.123:2882:2881;10.186.65.56:2882:2881' -l info -o 'obconfig_url=http://10.186.65.11:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sit,config_additional_dir=/data/1/sit/etc3;/data/log1/sit/etc2,cluster_id=16777777,datafile_disk_percentage=80,cpu_count=16,system_memory=5G'

Key parameters: -i: network interface (check with ifconfig) -p: service port (usually 2881) -P: RPC port (usually 2882) -n: cluster name -z: zone -d: cluster data directory -r: rootservice list -l: log level (default INFO) -o: additional startup options (including datafile_disk_percentage)

4.4 Add Server to Cluster

alter system add server '10.186.65.56:2882' zone 'zone3';

The OCP cluster overview page will refresh to show the new OBServer.

4.5 Other Replica Operations

Repeat the above steps for each tenant replica, offline/online other OBServers, and add missing replicas. After completion, the block_file under /data/1 is successfully shrunk.

4.6 Restart Cluster

Finally, restart the cluster to verify normal operation.

3 Summary

The datafile shrink operation effectively reinstalls OBServer on each node, which carries risk in production; therefore, backup beforehand. In similar failure cases, first check for alternative disk space (e.g., NFS) to store core files. Parameters datafile_disk_percentage and datafile_size can be increased dynamically without restarting the cluster, but decreasing them has no effect.