Analyzing OceanBase Freeze Dump Process via Log Parsing

1 T1002_Occam

1.1 Thread Introduction

The freeze‑check thread runs a check every 2 seconds; when a freeze is required it creates a checkpoint task that is processed by the freeze thread. Its operation can be verified in the logs by searching for the phrase “tenant freeze timer task”.

1.2 Log Flow

When a freeze is needed the system logs “[TenantFreezer] A minor freeze is needed”. The trigger condition is that the tenant’s active_memstore_used_ exceeds the memstore_freeze_trigger threshold. After the trigger, the system scans the tenant’s log stream and submits the corresponding freeze task to the freeze thread.

succeed to start ls_freeze_task(ret=0, ls_id={id:xxx})

2 T1002_LSFreeze

2.1 Thread Introduction

This thread moves freeze‑checkpoints that satisfy the flush condition from new_create_list to prepare_list . It uses the methods road_to_flush and ready_for_flush_ to decide whether a memtable’s rec_scn is frozen or has replay references.

Note: When a memtable is initialized, its associated freeze checkpoint is registered in a bidirectional list named new_create_list . The implementation can be found in ObTabletMemtableMgr::create_memtable() .

2.2 Log Flow

The logs do not show every internal step, but the presence of the message “road_to_flush end” indicates that the freeze process has completed.

[2023-08-18 06:44:51.285827] INFO  [STORAGE] road_to_flush (ob_data_checkpoint.cpp:333) [1553][T1002_LSFreeze1][T1002][Y0-0000000000000000-0-0] [lt=7] [Freezer] road_to_flush begin(ls_->get_ls_id()={id:1001})
[2023-08-18 06:44:51.285846] INFO  [STORAGE] road_to_flush (ob_data_checkpoint.cpp:341) [1553][T1002_LSFreeze1][T1002][Y0-0000000000000000-0-0] [lt=16] [Freezer] new_create_list to ls_frozen_list success(ls_->get_ls_id()={id:1001})
[2023-08-18 06:44:51.285861] INFO  [STORAGE] road_to_flush (ob_data_checkpoint.cpp:345) [1553][T1002_LSFreeze1][T1002][Y0-0000000000000000-0-0] [lt=3] [Freezer] ls_frozen_list to active_list success(ls_->get_ls_id()={id:1001})
[2023-08-18 06:44:51.285867] INFO  [STORAGE] road_to_flush (ob_data_checkpoint.cpp:355) [1553][T1002_LSFreeze1][T1002][Y0-0000000000000000-0-0] [lt=6] [Freezer] active_list to ls_frozen_list success(ls_->get_ls_id()={id:1001})
[2023-08-18 06:44:51.337395] INFO  [STORAGE] road_to_flush (ob_data_checkpoint.cpp:358) [1553][T1002_LSFreeze1][T1002][Y0-0000000000000000-0-0] [lt=16] [Freezer] road_to_flush end(ls_->get_ls_id()={id:1001})

3 T1002_Flush

3.1 Thread Introduction

The Flush thread runs every 5 seconds; its status can be identified by the log entry “traversal_flush timer task”. It iterates the checkpoints in prepare_list and creates an ObTabletMiniMergeDag object to execute as a DAG task.

3.2 Log Flow

For each data tablet (example tablet ID 200001), a DAG is created and added to the task queue.

[2023-08-18 06:44:51.335124] INFO  [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3377) [1655][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=29] add dag success(dag=0x7fa95f358b20, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7887337314793470841, dag_cnt=23, dag_type_cnts=22)
[2023-08-18 06:44:51.335132] INFO  [COMMON] create_and_add_dag (ob_dag_scheduler.h:1119) [1655][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7fa95f358b20)

If the DAG creation succeeds, the log records “schedule tablet merge dag successfully” and marks the task type as “MINI_MERGE”.

[2023-08-18 06:44:51.335134] INFO  [STORAGE.TRANS] flush (ob_memtable.cpp:2095) [1655][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=2] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1001}, tablet_id:{id:200001}, report_:null, for_diagnose:false, …})

4 T1002_DagScheduler

4.1 Thread Introduction

Based on the task type in the DAG queue, the system creates a thread named “T1002_MINI_MERGE” to execute the dump tasks. The first task created is ObTabletMergePrepareTask , which subsequently spawns ObTabletMergeTask and ObTabletMergeFinishTask .

4.2 Log Flow

In the “T1002_DagScheduler” thread, the tablet_id field is used to filter logs. Entries with type “DAG_MINI_MERGE” contain the corresponding task_id (e.g., YB427F000001-0006032C0D448715-0-0).

[2023-08-18 06:44:51.420180] INFO  [SERVER] add_task (ob_sys_task_stat.cpp:142) [1597][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] succeed to add sys task(task={start_time:1692341091420175, task_id:YB427F000001-0006032C0D448715-0-0, task_type:3, svr_ip:"127.0.0.1:2882", tenant_id:1002, is_cancel:false, comment:"info=DAG_MINI_MERGE;ls_id=1001;tablet_id=200001;…"})
[2023-08-18 06:44:51.420192] INFO  [COMMON] schedule_one (ob_dag_scheduler.cpp:2997) [1597][T1002_DagSchedu][T1002][YB427F000001-0006032C0D448715-0-0] [lt=12] schedule one task(task=0x7fa9264c8080, priority:"PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL)

5 T1002_MINI_MERGE

5.1 Thread Introduction

This thread executes the tasks scheduled by the “T1002_DagScheduler”.

5.2 Log Flow

By filtering the logs for the specific task_id , three tasks are observed: ObTabletMergePrepareTask , ObTabletMergeTask , and ObTabletMergeFinishTask .

5.2.1 ObTabletMergePrepareTask

The prepare task performs initialization and checks before the actual merge.

[2023-08-18 06:44:51.420180] INFO  [SERVER] add_task (ob_sys_task_stat.cpp:142) … task_id:YB427F000001-0006032C0D448715-0-0 …
[2023-08-18 06:44:51.420192] INFO  [COMMON] schedule_one (ob_dag_scheduler.cpp:2997) … task=0x7fa9264c8080 …

5.2.2 ObTabletMergeTask

The merge task writes macro blocks, consolidating multiple versions of records into a single record.

[2023-08-18 06:44:51.875958] INFO  [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:1555) … merge macro blocks ok(idx_=0, task={…})

5.2.3 ObTabletMergeFinishTask

The finish task creates a new MINI SSTable and releases the associated memtables.

[2023-08-18 06:44:51.889896] INFO  [STORAGE] release_memtables (ob_i_memtable_mgr.cpp:164) … succeed to release memtable(ret=0, i=1, scn={val:1692341091275445526, v:0})
[2023-08-18 06:44:51.890015] INFO  [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2563) … dag finished(dag_ret=0, runtime=469823, dag_cnt=9, …)

After the DAG finishes, all related tasks are cleared, marking the successful completion of the data freeze and dump process.