DynamicTp: Meituan's Dynamic Thread Pool Framework for Runtime Parameter Tuning and Monitoring

Background and Pain Points

When using ThreadPoolExecutor, developers face three common pain points: (1) difficulty determining optimal core parameters (corePoolSize, maximumPoolSize, queueCapacity, etc.); (2) parameter adjustments require code changes and service restarts; (3) thread pools operate as black boxes with no runtime visibility until failures occur.

The article notes that ThreadPoolExecutor already provides set/get methods for most core parameters, enabling runtime modification. Since most internet projects use microservices with distributed configuration centers (highly available), combining configuration centers with these setter methods allows dynamic parameter updates without restarts.

DynamicTp Framework Overview

DynamicTp (v1.0.9) extends ThreadPoolExecutor to achieve three goals: dynamic parameter modification with immediate effect, real-time monitoring with alerting pushed to workplace platforms (DingTalk, WeCom, Feishu, Email), and periodic metric collection for visualization via Grafana.

Key features include:

Zero code intrusion : all configuration resides in the configuration center.

Lightweight : Spring Boot starter integration completed in 4 steps within ~3 minutes.

High extensibility : core functions expose SPI interfaces for custom implementations (config center, config parsing, notification, metric collection, task wrapping).

Production-proven at scale : references Meituan's internal thread pool practices.

Multi-platform alerts : supports config-change notifications, liveness alerts, queue-capacity thresholds, rejection-policy triggers, task-execution timeouts, and task-wait timeouts; each alert format shows interval count / cumulative total (A/B).

Monitoring outputs : JsonLog (disk), MicroMeter (Grafana dashboards), and custom Endpoint ( dynamic-tp HTTP access); all extensible via SPI.

Task enhancement : TaskWrapper interface with built-in MdcTaskWrapper, TtlTaskWrapper, SwTraceTaskWrapper for context propagation (MDC, TransmittableThreadLocal, SkyWalking).

Compatibility : JUC thread pools and Spring's ThreadPoolTaskExecutor become manageable by adding @DynamicTp on @Bean definitions.

Reliability : framework thread pools implement Spring lifecycle callbacks to drain queues before container shutdown.

Multi-mode : provides EagerDtpExecutor (referencing Tomcat's thread pool) for IO-intensive scenarios.

Multi-config-center support : Nacos, Apollo, Zookeeper, Consul, Etcd out of the box; SPI for others.

Middleware thread pool management : integrates Tomcat, Jetty, Undertow, Dubbo, RocketMQ, Hystrix, gRPC for unified parameter tuning and monitoring.

Architecture Design

Code Structure

The project comprises seven modules:

adapter : adapts third-party component thread pools (Tomcat, Jetty, Undertow, Dubbo, RocketMQ, Hystrix, gRPC).

common : shared utilities and decoupled dependencies.

core : core logic — dynamic parameter adjustment, monitoring, alerting, and workflow orchestration.

example : usage demonstration.

extension : optional extensions (Redis rate-limiting, email sending, SkyWalking context propagation).

logging : internal logging configuration, primarily for metric log output.

starter : auto-configuration and dependency encapsulation for each functional module.

Configuration Change Listening Module

Listens to designated config files in supported configuration centers (Nacos, Apollo, Zookeeper, Consul, Etcd); extensible via SPI.

Parses configuration content with built-in YML, Properties, JSON parsers; extensible via SPI.

Notifies the thread pool management module to refresh parameters.

Service Internal Thread Pool Management Module

On startup, pulls configuration from the config center, creates thread pool instances, and registers them in the internal registry and Spring container.

Receives refresh events from the listening module and updates thread pool parameters dynamically.

Application code obtains thread pool instances via dependency injection (recommended) or DtpRegistry.getDtpExecutor(name).

Third-Party Component Thread Pool Management

At startup, acquires thread pools from common middleware (Tomcat, Jetty, Undertow, Dubbo, RocketMQ, Hystrix, gRPC) and places them under framework management.

Handles parameter refresh, metric collection, and alert events for these managed pools.

Monitoring Module

Collects and exports metrics via three default mechanisms (all SPI-extensible):

JsonLog : writes structured JSON logs to disk for external collection/parsing.

MicroMeter : with MicroMeter dependencies, exposes endpoints for Prometheus/Grafana integration.

Endpoint : custom HTTP endpoint ( dynamic-tp) for real-time metric access.

Notification & Alert Module

Integrates with workplace platforms (DingTalk, WeCom, Feishu, Email); extensible via SPI. Alert types:

Thread pool main parameter change notifications.

Blocking queue capacity reaching configured threshold.

Thread pool liveness reaching configured threshold.

Rejection policy triggered (format A/B: interval count / cumulative total).

Task execution timeout (format A/B).

Task queue wait timeout (format A/B).

Project References

Gitee: https://gitee.com/dromara/dynamic-tp GitHub:

https://github.com/dromara/dynamic-tp