Troubleshooting OceanBase Single‑Node Replica Expansion and Log Disk Size Issues

The author, a DBA from SuShang Bank, reports a recent OceanBase test where a single‑node replica expansion was performed and numerous problems were encountered, prompting a comprehensive analysis.

1. Tenant Expansion – The test revealed that the sys tenant must be expanded first; otherwise the cluster becomes a “pseudo‑cluster”. The required commands are:

ALTER RESOURCE POOL sys_pool ZONE_LIST=('zone1','zone2','zone3');
ALTER TENANT sys LOCALITY='F@zone1,F@zone2';
ALTER TENANT sys LOCALITY='F@zone1,F@zone2,F@zone3';

2. Table Replica Management – OceanBase 4.X automates table replica placement based on PRIMARY_ZONE . Older syntax such as:

ALTER TABLE test_t1 set locality = 'F@zone3,F@zone2,F@zone1';
ALTER TABLE test_t1 locality='F,R{all_server}@ZONE1, F,R{all_server}@ZONE2, F,R{all_server}@ZONE3';

fails with syntax errors in newer versions.

3. Log Disk Size Misconfiguration – The primary failure stemmed from the log_disk_size setting (2 GB). After tenant creation, the system could not allocate enough PALF instances because the required minimum is (zone_count+1) × 512 MB = 2048 MB . Queries used to diagnose the issue include:

SELECT tenant_name,database_name,table_name,partition_name,zone,svr_ip,role FROM OCEANBASE.CDB_OB_TABLE_LOCATIONS a JOIN oceanbase.DBA_OB_TENANTS b ON a.tenant_id=b.tenant_id WHERE DATABASE_NAME='MYDB' AND TENANT_NAME='mysql_tenant1';
SELECT zone,svr_ip,role,count(1) FROM OCEANBASE.CDB_OB_TABLE_LOCATIONS a JOIN oceanbase.DBA_OB_TENANTS b ON a.tenant_id=b.tenant_id WHERE DATABASE_NAME='MYDB' AND TENANT_NAME='mysql_tenant1' GROUP BY zone,svr_ip,role;

Additional checks on log usage:

SELECT SVR_IP,SVR_PORT,UNIT_ID,TENANT_ID,CEIL(LOG_DISK_SIZE/1024/1024) LOG_DISK_SIZE_M, CEIL(LOG_DISK_IN_USE/1024/1024) LOG_DISK_IN_USE_M, ROUND(LOG_DISK_IN_USE/LOG_DISK_SIZE*100) LOG_USE_PCT, STATUS FROM oceanbase.GV$OB_UNITS ORDER BY SVR_IP,TENANT_ID;

4. Log Stream Fundamentals – OceanBase uses Log Streams (LS) to synchronize state across replicas. Each tablet maps to a log stream, which follows a Multi‑Paxos consensus. Log files reside under <data_dir>/clog/tenant_xxxx , with a default size of 64 MB per file. The log_disk_size and log_disk_percentage parameters control total allocation, and the system pre‑allocates space based on these values.

5. Debugging PALF Creation – Stack traces show the failure occurs in PalfEnvImpl::check_can_create_palf_handle_impl_() , where the check (zone_count+1) * MIN_DISK_SIZE_PER_PALF_INSTANCE <= log_disk_usage_limit_size_ evaluates to false because the tenant only has 1536 MB available after reserving the meta tenant’s 512 MB.

bool PalfEnvImpl::check_can_create_palf_handle_impl_() const {
  int64_t count = palf_handle_impl_map_.count();
  const PalfDiskOptions disk_opts = disk_options_wrapper_.get_disk_opts_for_recycling_blocks();
  bool_ret = (count + 1) * MIN_DISK_SIZE_PER_PALF_INSTANCE <= disk_opts.log_disk_usage_limit_size_;
  return bool_ret;
}

6. Summary and Recommendations – To successfully create LS for a tenant with three zones, the log_disk_size must be at least 2560 MB (or satisfy (zone_count+2)×512 MB ). Either increase the tenant’s log_disk_size or omit the parameter to let OceanBase allocate the default three‑times‑memory size. Ensuring sufficient log space prevents LS creation failures and uneven leader distribution across zones.