Analysis of OBLogProxy Failure in Binlog Mode and Its Resolution

1. Background

In a customer environment, an OBLogProxy service that had not been used was manually restarted after a host expansion. The service started successfully, but the show binlog status command returned a pid of 0 and no new Binlog entries for the past three months.

The service start log was:

[root@oblogproxy oblogproxy]# ./run.sh start
DEPLOY_PATH : /usr/local/oblogproxy
is_running : (6347)/usr/local/oblogproxy logproxy is running!
kill_proc : (6347)/usr/local/oblogproxy succ!
./run.sh: line 53:  6347 Terminated              ./bin/${BIN} -f ./conf/conf.json &>${log_path}/out.log
is_running : (7735)/usr/local/oblogproxy logproxy is running!
logproxy started

2. Fault Analysis

2.1 OBLogProxy Status Check

Directly connect to the OBLogProxy service and run:

mysql -h 'IP' -P 2983
show  Binlog  instances\G

The result shows three columns— running , obcdc_running , and convert_running —indicating the status of the Binlog main process, the obcdc process, and the converter process respectively. All three must be Yes for normal operation.

Further checking the Binlog converter (bc) process revealed it was not started, prompting a log investigation.

2.2 Log Analysis

2.2.1 BC Process Log

The bc process log file is oblogproxy/run/$cluster_name/$tenant_name/log/binlog_converter.log :

[2025-02-24 10:33:14] [critical] obcdc_access.cpp(96): Failed to init libobcdc, ret: -4018
[2025-02-24 10:33:14] [error]  binlog_converter_entry.cpp(66): !!! Exit  binlog converter process with pid: 24479
[2025-02-24 10:33:14] [warning] thread.cpp(37): The current thread :(24482) has already stopped
[2025-02-24 10:33:14] [warning] thread.cpp(37): The current thread :(24482) has already stopped

The bc process exits immediately because libobcdc fails to initialize with error code 4018.

2.2.2 libobcdc Log

The libobcdc log file is oblogproxy/run/$cluster_name/$tenant_name/log/libobcdc.log :

[2025-02-24 10:33:11.551061] WDIAG [TLOG] get_data_dict_in_log_info (ob_log_meta_data_queryer.cpp:49) … errcode=-4018 can't find datadict snapshot_scn older than start_timestamp_ns(ret=-4018, ret="OB_ENTRY_NOT_EXIST", tenant_id=1006, start_timestamp_ns=1733201736158282000)
[2025-02-24 10:33:11.551097] WDIAG [TLOG] get_data_dict_in_log_info_ (ob_log_meta_data_service.cpp:375) … errcode=-4018 sql_queryer get_data_dict_in_log_info fail(ret=-4018, ret="OB_ENTRY_NOT_EXIST", tenant_id=1006, start_timestamp_ns=1733201736158282000, data_dict_in_log_info={snapshot_scn:-1, end_scn:-1, start_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}})
[2025-02-24 10:33:11.551149] ERROR issue_dba_error (ob_log.cpp:1875) … errcode=-4018, file="ob_log_meta_data_service.cpp", line_no=389, info="[FATAL][DATA_DICT] Can't find suitable data_dict to launch OBCDC, please try use online schema(refresh_mode=online && skip_ob_version_compat_check=1)"

Searching for the keyword error reveals the message "can't find datadict snapshot_scn older than start_timestamp_ns" , indicating that the downstream request timestamp 1733201736158282000 has no corresponding consumable Clog records.

By logging into the sys tenant and querying the minimum consumable positions of all log streams, we obtain:

SELECT CEIL(MAX(BEGIN_SCN)/1000) AS START_TS_US FROM oceanbase.GV$OB_LOG_STAT;
+------------------+
| START_TS_US      |
+------------------+
| 1737628799517391 |
+------------------+

SELECT CEIL(MAX(START_SCN)/1000) AS START_TS_US FROM oceanbase.DBA_OB_ARCHIVELOG;
+------------------+
| START_TS_US      |
+------------------+
| 1733387084685441 |
+------------------+

Comparing the request timestamp with the minimum consumable position and the archive start time shows that the request timestamp is earlier than both, causing the bc process to fail.

3. Conclusion

After restarting OBLogProxy, it attempted to pull data starting from 2024‑12‑03, but the archive start time is 2024‑12‑05, so the service cannot find upstream consumable data and aborts.

4. Resolution

Release and rebuild the Binlog task. Delete all existing Binlog files for the tenant and recreate the task; the binlog_converter process will then pull and parse Clog from the current time.

1. Log in to the Binlog Server.
   mysql -h $IP -P2983

2. Release the Binlog task.
   DROP BINLOG FOR TENANT $cluster_name.$tenant_name;

3. Create a new Binlog task.
   CREATE BINLOG FOR TENANT $cluster_name.$tenant_name TO USER $username PASSWORD `xxx` WITH CLUSTER URL '$obconfig_url';

4. Verify all Binlog instance statuses.
   SHOW BINLOG INSTANCES;

Expected status: the columns running, obcdc_running and convert_running should all be yes.

5. Further Reading

How to Determine Whether the Startup Timestamp Is Too Small? [2]

How to Rebuild a Binlog Task? [3]

6. References

[1] oblogproxy documentation: https://www.oceanbase.com/docs/community-oblogproxy-doc-1000000000861451

[2] How to determine whether the startup timestamp is too small?: https://www.oceanbase.com/docs/common-oceanbase-database-cn-1000000002013720

[3] How to rebuild a Binlog task?: https://www.oceanbase.com/docs/community-oblogproxy-doc-1000000001999434