How to Diagnose and Fix OceanBase Tenant Memory Exhaustion Issues

1. Background

1.1 Phenomenon Description

Users reported an error when accessing the OceanBase database:

ErrorCode = 4013, SQLState = HY001, Details = No memory or reach tenant memory limit

. Restarting the OBServer temporarily resolves the issue, but the error reappears after a while.

1.2 Environment Information

Database version: Enterprise Edition 4.3.3.1

Deployment architecture: 2-2-2

Tenant mode: MySQL

2. Fault Diagnosis

2.1 Obtain TRACE_ID of the Erroneous SQL

Developers use ODC to execute the query, which also returns the same error. The ODC execution record shows the SQL's TRACE_ID.

Clicking the TRACE_ID opens a page where you can select "Full‑link Diagnosis". Hovering over com_query_process shows the SVR_IP field (example shown for a local environment).

For methods to obtain executed SQL's TRACE_ID and SVR_IP, see the references at the end.

2.2 Filter Detailed Logs by TRACE_ID

Using the error time range and TRACE_ID, filter OBServer logs on the SVR_IP node (node 113). The logs reveal that tenant memory usage plus newly requested memory exceeds the tenant limit: tenant_hold+alloc_size>tenant_limit #[OOPS]: alloc failed reason is that tenant memory has reached the upper limit(tenant_id: 1010, tenant_hold: 11595448320, tenant_limit: 11596411700, alloc_size: 2097152)

2.3 View Memory Usage of Each Context in the Tenant

Filtering the OBServer logs on node 113 for tenant 1010 shows that the DEFAULT_CTX_ID context consumes the most memory, approaching the tenant's memory limit.

# grep 'malloc_allocator.*tenant: 1010' observer.log.20250522105401599 -A 20

2.4 Check Memory Usage of Each Mod in the Specified Context

On node 113, the SqlSessionSbloc mod occupies the most memory, using over 80% of the tenant's memory. Continuous growth eventually exhausts tenant memory.

2.5 Diagnose Continuous Growth of Mod Memory Consumption

Since OceanBase 4.1.0, the virtual table __all_virtual_malloc_sample_info records key memory allocation information. When a module allocates memory exceeding GB, this table helps locate the root cause.

Relevant fields of __all_virtual_malloc_sample_info:

Column

Type

Description

svr_ip

varchar:MAX_IP_ADDR_LENGTH

IP address

svr_port

int

Port number

tenant_id

int

Tenant ID

ctx_id

int

CTX ID

mod_name

varchar:OB_MAX_CHAR_LENGTH

Module name

back_trace

varchar:DEFAULT_BUF_LENGTH

Allocation stack trace

ctx_name

varchar:OB_MAX_CHAR_LENGTH

CTX name

alloc_count

int

Number of allocations

alloc_bytes

int

Total allocated bytes

2.6 Query Memory Allocation Info for a Specific Mod

Note: The virtual table __all_virtual_malloc_sample_info can only be queried in the sys tenant on x86_64 architectures.

select * from oceanbase.__all_virtual_malloc_sample_info where mod_name='SqlSessionSbloc' order by alloc_bytes DESC;

2.7 Convert Back Trace to Stack Trace

# addr2line -pCfe bin/observer 0x1f5951a0 0x788cfac 0x7ce43e0 0x7ce3a48 0x15111873 Oxf35e3e8 0x79304ee 0x792e9af 0x791e66b 0xf054727 Ox1f6509be

The stack analysis shows that the function set_login_info directly allocates memory, suggesting a leak in oceanbase::sql::ObSQLSessionInfo::set_login_info.

2.8 Verify Code to Confirm the Issue

OceanBase Community Edition runs in MySQL tenant‑compatible mode, so the open‑source code can be inspected.

Open the OceanBase Community repository, search for ObSQLSessionInfo , and locate ob_sql_session_info.cpp. Use the "Blame" view to find changes related to set_login_info.

Inspect the commit titled "fix memory leak caused by calling set_login_info() when reusing a session". The bug was fixed in version 4.3.5 and later.

If the fix is not found, developers should further analyze the set_login_info function logic.

2.9 Fault Resolution

After confirming with the development team, upgrade the database to version 4.3.5 (or later). Post‑upgrade, the problematic SqlSessionSbloc mod disappears from the top memory‑consuming list, and the tenant memory exhaustion issue is resolved.

SELECT svr_ip, ctx_name, mod_name, SUM(hold)
FROM oceanbase.__all_virtual_memory_info
WHERE tenant_id = 1010
GROUP BY svr_ip, ctx_name, mod_name
ORDER BY SUM(hold) DESC
LIMIT 10;

3. Summary

This article demonstrates how to trace tenant memory exhaustion in OceanBase by analyzing OBServer logs, identifying the memory‑heavy mod, using the __all_virtual_malloc_sample_info virtual table for deeper insight, and finally applying a version upgrade to fix the underlying memory leak.

References

[1] ODC (OceanBase Developer Console): https://www.oceanbase.com/product/odc

[2] Method to obtain SQL execution SVR_IP and TRACE_ID: https://www.oceanbase.com/knowledge-base/oceanbase-database-1000000000225641

[3] Using __all_virtual_malloc_sample_info to diagnose memory module issues: https://www.oceanbase.com/knowledge-base/oceanbase-database-1000000002393827

[4] OceanBase Community Edition source code (v4.3.5): https://github.com/oceanbase/oceanbase/tree/v4.3.5_CE_BP2_HF1