StarRocks THRIFT_EAGAIN Timeout: Root Cause Analysis & Fix

Background

StarRocks write tasks failed with the error "FE RPC failure, reason=THRIFT_EAGAIN (timed out)", while data queries remained unaffected.

Observations

A traffic spike occurred around 09:10, after which write traffic dropped to zero.

JVM used memory consistently stayed below committed memory.

JVM Memory Concepts: Committed vs Used

1. Committed Memory

Meaning : Memory the JVM has successfully obtained from the OS and promised to use. This memory is reserved for the JVM and represents the actual usable upper limit.

Role : Reflects system resources occupied by the JVM. When more memory is needed, the JVM first uses committed memory; if insufficient, it requests more from the OS up to the maximum limit ( -Xmx).

Example : With -Xmx512m -Xms256m, initial committed memory is 256 MB. It can grow up to 512 MB as the JVM runs.

2. Used Memory

Meaning : Memory actually holding objects and data. It is a subset of committed memory.

Role : Helps monitor memory usage efficiency and load. If used memory approaches committed memory, garbage collection may trigger, potentially leading to OutOfMemoryError.

Example : If committed is 256 MB and used reaches 200 MB, 56 MB remains available.

Relationship

used ≤ committed . Monitoring both helps with performance tuning and troubleshooting; when used nears committed, consider increasing -Xmx or optimizing application memory usage.

Problem Analysis

Monitoring showed JVM usage frequently spiking to 80% then dropping. Daytime workloads involved many full writes preceded by TRUNCATE TABLE operations, causing intensive object manipulation and data updates in the JVM.

Protocol Analysis: Thrift and THRIFT_EAGAIN

What is THRIFT_EAGAIN?

Thrift is a distributed service framework for RPC. THRIFT_EAGAIN (timed out) on Unix indicates "resource temporarily unavailable", often related to network connections or retry mechanisms.

Typical Trigger Scenarios

Client sends request : Server busy processing other requests, cannot respond immediately, client read times out.

Server processes request : Reading client data, but data not fully arrived (e.g., network latency causing incomplete fragmented transmission).

Connection pool exhausted : Client or server connection pool reaches limit, new connection requests rejected.

TRUNCATE TABLE Resource Consumption Analysis

1. Metadata Updates

Principle : TRUNCATE TABLE deletes all data but keeps table structure. The FE updates metadata (stored in MySQL or RocksDB) to record the table as empty.

Resource Cost : CPU for metadata parsing and updates; disk I/O for persisting metadata changes.

2. Data File Deletion

Principle : Data files on BE nodes are deleted.

Resource Cost : Significant disk I/O, especially with many or large files; network bandwidth for FE to send delete commands to BEs.

3. Memory Release

Principle : Data cached in BE memory buffers is released.

Resource Cost : Minor CPU for memory management operations (e.g., updating allocation tables).

4. Concurrency Control and Locking

Principle : Locks on the table or related resources prevent concurrent reads/writes during truncation.

Resource Cost : CPU for lock acquisition, release, and conflict detection; potential blocking of other operations, reducing concurrency.

Solutions

1. Adjust thrift_rpc_timeout_ms

This parameter sets the maximum time (in milliseconds) a Thrift RPC client waits for a server response. It prevents indefinite blocking and improves system robustness by allowing timeouts, retries, and error handling.

Example configuration in fe.conf: thrift_rpc_timeout_ms = 3000 Sets timeout to 3000 ms (3 seconds). Too short causes false timeouts; too long degrades responsiveness. Must be tuned to network conditions and server capacity. Note: different components may use different parameter names (e.g., thrift_timeout, rpc_timeout).

2. Adjust Write Frequency and Table Model

Batch writes (accumulate before writing).

For frequently updated tables, use incremental writes instead of TRUNCATE.

For frequently updated tables, change table model to Primary Key model.

3. Increase FE Node Resources

Add CPU, memory, or disk to FE nodes.

4. Separate Real-time and Offline Workloads

Identify frequently updated tables, build a new cluster, and migrate "near-real-time" and real-time tables to achieve separation of offline and real-time data warehouses.

Execution and Results

Emergency Recovery

Restarted the FE leader node.

Post-Recovery Optimization

Changing table models and write frequency requires development effort and time. Adding hardware incurs cost. Cluster separation also takes time. Considering cost and efficiency, the immediate step was to increase thrift_rpc_timeout_ms from 5 seconds to 30 seconds: thrift_rpc_timeout_ms = 30000 After this adjustment, the timeout issue was alleviated.

Next Steps

Continue driving write-pattern refactoring. Long-term, separating real-time and offline data warehouses is considered more stable.