From Druid to MetaQ: A Developer’s Journey Through Middleware Mastery

Introduction

At Stanford, Steve Jobs delivered a memorable "connecting the dots" speech, inspiring the author to believe that current experiences will later form meaningful connections in life and work.

What does connecting the dots mean?

When one’s abilities are still growing, focusing on the present and doing the best within current limits eventually reveals unexpected connections.

1. Druid Connection Pool

In 2013 the author refactored a lottery award service from C# to Java, switching to Druid as the database connection pool. After deployment, performance improved tenfold, but the first daily request failed due to missing heartbeat configuration. Contact with the Druid author revealed the need for periodic heartbeat packets to keep Oracle connections alive.

This experience motivated deeper study of connection‑pool principles, including Druid’s array‑based implementation, Jedis’s Commons‑pool usage, and Netty’s FixChannelPool.

Conceptual: seeking help becomes addictive.

Technical: understanding connection‑pool implementation.

Architectural: client‑server communication requires heartbeat threads.

2. Cobar Sharding Middleware

Also in 2013 the author discovered Cobar, an open‑source sharding middleware. Configuring Cobar with Navicat allowed a single MySQL‑like connection while data was evenly distributed across multiple databases, feeling like a dimensional breakthrough.

To grasp sharding principles, three months were spent copying Cobar’s core code, learning NIO networking, Maven assembly packaging, and memory‑pool concepts later applied in Netty.

The knowledge proved valuable at Ele.me, where the author quickly understood the architecture of a distributed database and contributed a bug fix to its transaction handling.

3. MetaQ Message Queue

In 2015 the author joined Shenzhou Dache and faced performance bottlenecks. MetaQ, an open‑source MQ, uses a cluster‑consumer model similar to Kafka, allowing one message to be processed by a single consumer within a group while multiple groups can consume the same message.

MetaQ was used for order‑status notifications, broadcasting messages to order, big‑data, and dispatch subsystems. Different consumer groups stored offsets in ZooKeeper, demonstrating the elegance of MQ broadcasting.

Business system pushes messages to MetaQ.

TCP gateway consumes broadcast messages and forwards them to apps.

All servers receive the broadcast and push to connected sessions.

4. Task Scheduling Systems

By 2018 a reliable scheduler was needed. The author first examined XXL‑Job, then learned from Meituan’s Crane and Alibaba Cloud’s ScheduleX, eventually building a custom scheduler that replaces Quartz with a time‑wheel and uses RPC‑based push.

The system works but still faces database bottlenecks at high task volumes and concerns about container deployment.

Conclusion

Inspired by Steve Jobs’ speech, the author believes that loving one’s work brings true fulfillment. Continuing to study source code, share knowledge, and embody dedication can help fellow programmers on their own journeys.