Building a Redis‑Based Distributed Queue to Cut HBase IO Bottlenecks

1. My View on Good Code

1.1 What is Good Code

If you have read "Design Patterns", you may think code that follows the six SOLID principles is good; if you have read "Clean Code", you may think cleanliness is the standard; if you study Spring source code, you may value clever design, high abstraction, and flexible configuration. In short, good code is subjective.

1.2 My Definition of Good Code

Based on my current understanding, good code has the following characteristics:

Usability

Good code must be usable; it should work. Beautiful tricks without functionality are meaningless.

Readability

Code has two audiences: the compiler and the maintainer. As long as it follows syntax it satisfies the compiler, but it must also be understandable to future developers.

Other Excellent Traits

Maintainability

Scalability

Reusability

Robustness

Testability

3. Let Code Dance on the Computer

The purpose of writing code is to convey our thoughts to the computer so it can perform tasks. Good code not only fulfills the intent but does so quickly, efficiently, and completely.

II. Why We Needed a Solution

During the May 2020 holiday, an HBase alarm indicated IO pressure near the bottleneck, affecting read/write performance. To prevent similar issues during traffic spikes, we designed a lightweight Redis‑based distributed consumption queue to shard data, batch‑deduplicate, and write to HBase+Solr, thereby reducing IO pressure.

III. How We Built It

Overall Design

The component consists of three core modules: master (coordinator), writer (data‑writing node), and worker (processing node). It uses a weak‑central mechanism where any configured node can become master, writer, or worker, providing high availability without a single point of failure.

Master Responsibilities

Detect live workers and their status.

Assign task queues to alive workers.

Monitor the load of the Redis queues in real time.

Writer Responsibilities

Shard real‑time write tasks into different queues.

Monitor the load of its own queues.

Worker Responsibilities

Report its status and keep a heartbeat.

Consume and process the data assigned to it on a schedule.

“Talk is cheap, show me the code.” – a great Linux guru

IV. What We Delivered

Component Package Diagram

Clean Code Structure

Clear comments describing class responsibilities.

Configurable startup options.

Use of lambda expressions to reduce boilerplate.

Assertions replace traditional if‑else for readability.

The project contains 60 classes, 1,623 lines of core code, averaging 27 lines per class; the largest class is under 200 lines.

Strong Extensibility

Hook‑callback design allows users to inject custom logic quickly, enabling rapid business extension.

Runtime Log Screenshots

Log File

Master Queue Assignment Log

Worker Data Consumption Log

Writer Queue Load Monitoring Log

Redis Consumption Dashboard

V. Our Gains

After deployment, HBase service metrics showed overall usage reduced to about 50% of the previous level.

HBase IOPS Monitoring

HBase CPU Usage Monitoring

VI. Outlook

Package the Redis‑based queue as an independent Spring‑Boot‑Starter for high reusability and extensibility.

Apply the component to various distributed task‑queue scenarios such as task centers.

Open‑source the project in the future.

VII. My Reflections

Good code, like a well‑written article, should be pleasant to read. Different languages have different stylistic expectations—Python relies on indentation, C on procedural flow, Java on object‑oriented narratives—but readability and understandability are universal. Maintaining usability, readability, and maintainability ensures code runs longer and farther.