Analyzing OBServer Crash Logs with observer.log and gdb

1 Preparation Knowledge

Common reasons for OBServer process crash include program bugs, file corruption, bad disk blocks, or memory errors. The system automatically generates coredump files, which capture memory snapshots and thread stacks for debugging.

If a coredump is not generated, the observer.log can be used to obtain the stack trace.

The method described works for all OceanBase versions.

2 Analysis Steps

1. Locate Crash Log

Search observer.log for the keyword "CRASH ERROR". Example log entry:

CRASH ERROR!!! sig=11, sig_code=2, \
sig_addr=7f3edd31dffb, timestamp=1725496052323606, \
tid=57605, tname=TNT_L0_1002, \
trace_id=20970917872454-1707004480400037, \
extra_info=((null)), lbt=0x9baead8 \
0x9b9f358 0x7f43d58e562f \
0x7f43d52525fc 0x95eeda9 \
0x95ec568 0x95e6c0c \
0x95e4c33 0x9cbf4c7 \
0x93be9ee 0x939e320 \
0x93bd64e 0x939c105 \
0x939c6e6 0x2cff1c1 \
0x9918a74 0x9917461 0x9913f1e

Manual line breaks are added for readability.

2. Obtain Crash Thread Stack

Parse memory addresses with addr2line:

addr2line -pCfe /home/admin/oceanbase/bin/observer \
0x9baead8 0x9b9f358 0x7f43d58e562f 0x7f43d52525fc \
0x95eeda9 0x95ec568 0x95e6c0c 0x95e4c33 0x9cbf4c7 \
0x93be9ee 0x939e320 0x93bd64e 0x939c105 0x939c6e6 \
0x2cff1c1 0x9918a74 0x9917461 0x9913f1e

The stack is read from bottom to top; the fifth line indicates the crash location in ObMPStmtExecute::copy_or_convert_str .

3. Locate Exact Code Position

Use gdb (version ≥ 9.0) on the debug binary to map addresses to source lines, for example:

(gdb) list *0x95eeda9
0x95eeda9 is in oceanbase::observer::ObMPStmtExecute::copy_or_convert_str(oceanbase::common::ObIAllocator&, oceanbase::common::ObCollationType, oceanbase::common::ObCollationType, oceanbase::common::ObString const&, oceanbase::common::ObString&, long) (./src/observer/mysql/obmp_stmt_execute.cpp:1428).
(gdb) list *0x95ec568
0x95ec568 is in oceanbase::observer::ObMPStmtExecute::parse_basic_param_value(... ) (./src/observer/mysql/obmp_stmt_execute.cpp:1237).
(gdb) list *0x95e6c0c
0x95e6c0c is in oceanbase::observer::ObMPStmtExecute::parse_param_value(... ) (./src/observer/mysql/obmp_stmt_execute.cpp:1372).
(gdb) list *0x95e4c33
0x95e4c33 is in oceanbase::observer::ObMPStmtExecute::before_process() (./src/observer/mysql/obmp_stmt_execute.cpp:512).

4. Code Analysis

The crash occurs in ObMPStmtExecute::copy_or_convert_str at line 1428, where a MEMCPY copies from src.ptr() to buf + extra_buf_len . Signal 11 indicates an invalid memory access, confirming that src.ptr() is a null pointer.

5. Search Knowledge Base

Search the function name in the official knowledge base to find related bug reports; the bug matches the observed behavior where the execute protocol reads a null pointer before the send‑long‑data protocol finishes processing.

3 Summary

Following these five steps—locating the crash log, extracting the stack, mapping addresses with gdb, analyzing the offending code, and consulting the knowledge base—allows rapid identification of the root cause of an OBServer crash.