Design and Implementation of a Distributed Timer Service at Tencent

Timer (Timer) is a component frequently used in business development for delayed notification tasks. This article, based on the author’s practical experience at Tencent WXG backend, introduces how to quickly build a distributed timer service that meets common business needs and discusses solutions to problems encountered during implementation.

The article first defines the scenarios where timers are needed, such as automatically closing unpaid orders, periodically checking refund status, and sending activation SMS for newly created stores that have not uploaded products within a certain period.

It then explains the essence of a timer: a data structure where tasks with nearer deadlines have higher priority, supporting basic operations Add, Delete, Run, and optional Update. Two execution models are described – polling and sleep/wake based on the nearest deadline.

Key design goals include support for task submission, deletion, notification, reliable message delivery (at‑least‑once, aiming for exactly‑once), data persistence, high availability, and controllable latency.

The core data structure is a time wheel (including multi‑level time wheels). A time wheel is a circular structure divided into slots, each representing a time interval; tasks are placed into slots based on their deadline modulo the number of slots. The article shows diagrams of a single‑level and multi‑level time wheel.

Industry implementations are surveyed, including Redis ZSet, delayed‑queue‑enabled brokers (RabbitMQ, Beanstalkd, Tencent TDMQ), and time‑wheel‑based solutions.

The author’s solution stores tasks in a KV system (tablekv) using the execution timestamp as the shard key (bucket). A daemon periodically scans tables, extracts expired tasks, and pushes them to an event center. The event center forwards tasks to a broker, which notifies the business side. Architecture diagrams illustrate the Scheduler daemon, broker, and Notifier components.

Implementation challenges and their solutions are discussed:

Business isolation via biz_type and configurable rate limits.

Empty bucket inefficiency in the time wheel, mitigated by maintaining an ordered queue of non‑empty buckets.

Distributed rate limiting using Tencent’s rate‑limit component for both task insertion and notification.

Distributed single‑instance disaster recovery using a Chubby‑based lock to ensure only one daemon instance runs while providing failover.

Reliable delivery through persistent KV storage and an event center that guarantees at‑least‑once processing.

Timely delivery via multi‑threaded/ coroutine execution using Tencent’s BatchTask library.

Task expiration handling by setting KV TTL after successful notification.

Additional risk considerations such as clock drift and strong dependency on the event center.

The article concludes that the current timer service satisfies existing overseas payment needs but may require further optimization for higher load. It also includes a brief recruitment notice for the overseas payment team.