Graceful Shutdown of RocketMQ Consumers in Spring: A Real-World Debugging Tale

1

On a night in December 2018, I was about to release a feature and head home when an alarm sounded the moment I clicked the deploy button. The error count was low, so I assumed it was a transient issue and continued deploying the remaining machines.

2

Inspecting the logs revealed the following exception:

org.apache.ibatis.exceptions.PersistenceException: ### Error querying database. Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.alibaba.druid.pool.DataSourceClosedException: dataSource already closed

The message suggested that a closed data source was being accessed, which only happens when the process is terminated.

The error appears to be caused by a data source that has already been closed.

A data source is closed only when the process is killed.

Could the application shutdown be insufficiently graceful? Traffic is usually drained before a release, so it shouldn't happen.

While scrolling through the logs late at night, the word rocketmq caught my eye.

I realized that a RocketMQ consumer was still running when the application was shutting down. The external traffic had been cut off, but the consumer was not notified, causing it to throw an exception.

3

My shallow understanding of RocketMQ tells me that its consumer uses an ACK mechanism; if an error occurs, the message will be retried later, so no messages are lost. If the consumer code is idempotent, business logic remains unaffected. However, this consumer was not my code.

I glanced at the consumer’s source (not shown) and saw that it registers a ShutdownHook with the highest priority, invoking shutdown for graceful exit. In practice, that priority flag proved useless.

According to the article "ShutdownHook Principles", multiple ShutdownHooks—including Spring’s container shutdown—run concurrently without a defined order.

4

After understanding the root cause, I quickly wrote an interface to gracefully stop the RocketMQ consumer before the kill script executed, allowing me to leave work on time.

5

That night I dreamed my boss wanted every system refactored, which spurred me to rethink the solution. Implementing an interface that must be called from a stop script felt inelegant and non‑portable across projects.

Since the problem stemmed from the bean destruction order during Spring’s container shutdown, I considered using Spring’s depend‑on attribute to enforce the correct sequence.

Initially, the dependency graph looked like this:

Manually adding depend‑on to each bean in the XML seemed to work, but a second project revealed a far more complex graph:

With 26 beans, the dependency management became unwieldy, and I feared the refactor would take ages.

6

Later, I discovered the official RocketMQ Spring Boot starter on GitHub. Its implementation uses Spring’s SmartLifecycle interface, whose start method runs after all beans are initialized and whose stop method runs before bean destruction—perfect for graceful consumer shutdown.

The relevant code snippet is simply: SmartLifecycle I also revisited the Spring container shutdown flow in AbstractApplicationContext.doClose and summarized it in the diagram below:

The diagram shows that before bean destruction, Spring first closes lifecycle beans and then publishes a ContextClosedEvent. The official starter leverages the Lifecycle interface to hook into this sequence.

7

To present the options to my boss, I outlined two possible solutions:

Replace all custom shutdown logic with the official RocketMQ Spring Boot starter, which is maintained by the community but may involve high migration cost.

Listen for ContextClosedEvent and encapsulate graceful shutdown logic in a reusable component that downstream services can simply depend on.