From a Naïve Scheduled Task to Scalable Delayed‑Task Solutions for 10M+ Orders

Introduction

A candidate failed a second‑round interview at ByteDance when asked how to automatically cancel orders that remain unpaid for 30 minutes. He answered with a naive scheduled task that scans the whole table every minute.

Why a simple scheduled task (Cron) is a low‑level answer

In low‑concurrency, small‑data systems a Spring @Scheduled job works, but in high‑traffic scenarios it has three fatal drawbacks:

Timeliness : polling introduces delay and cannot achieve second‑level precision.

Database pressure : full‑table scans turn the data flow from push to pull, causing CPU spikes.

Resource waste : most minutes have no expired orders, yet the task still runs.

The high‑scoring answer must avoid polling the database and let expired orders “find their way” to the processor.

Core architectures: three mainstream solutions

Solution 1: Redis key‑expiration listener (a trap)

Some candidates suggest storing the order ID in Redis with a 30‑minute TTL and relying on the expiration event.

Unreliable : the expired event is “fire‑and‑forget”. If the service restarts or the network glitches, the event can be lost and the order never cancels.

High latency : Redis deletes keys lazily and periodically, so the actual removal may be delayed by minutes.

Solution 2: Redis ZSet + polling (recommended)

This is the most common generic solution, leveraging Redis sorted sets.

Principle : store the exact expiration timestamp in the ZSet Score and the order ID in the Value.

Production phase (order) : ZADD delay_queue <timestamp> <OrderId> Consumption phase (polling) : a background thread runs each second and executes ZRANGEBYSCORE delay_queue 0 <now> LIMIT 0 10 to fetch orders whose score ≤ current time.

Advantages : high performance (memory read/write) and second‑level accuracy.

Advanced pitfall : if a Lua script deletes the Redis entry but the business logic fails (e.g., service crash), the order is permanently lost.

Full‑score patch : introduce an ACK mechanism or a two‑phase process. The Lua script atomically moves the order ID from delay_queue to processing_queue. After successful business processing, the entry in processing_queue is deleted. A watchdog thread rescans processing_queue for stale tasks and retries, guaranteeing at‑least‑once delivery.

Solution 3: Message queue / time wheel (architect‑level)

When the data volume reaches billions, a single ZSet becomes a performance bottleneck.

A. Message queue (RocketMQ / RabbitMQ)

RocketMQ 4.x only supports fixed delay levels (1 s, 5 s … 30 min). RocketMQ 5.0 adds arbitrary delay, or you can fall back to Redis ZSet.

RabbitMQ’s native TTL + dead‑letter queue suffers “head‑of‑line blocking”. The rabbitmq_delayed_message_exchange plugin is required to avoid this issue.

B. Time wheel algorithm (Hashed Wheel Timer)

Imagine a clock with 60 slots; the pointer moves one slot per second. An order that expires in 30 minutes is placed in the slot (current + 1800). When the pointer reaches that slot, the order is triggered.

Advantages : pure memory operation, extremely efficient.

Drawbacks : memory is volatile; a restart loses the wheel state.

Large‑scale practice : persist tasks in Redis ZSet for up to an hour, and load recent tasks into an in‑memory time wheel on service start for high‑frequency triggering.

Final “defence” checklist

Q1: Multiple nodes poll ZSet – how to avoid duplicate cancellations?

Answer: use a Lua script to make ZRANGE and ZREM atomic, so only the node that successfully removes the entry proceeds. Also make the cancel service idempotent.

Q2: Redis ZSet becomes a huge key (tens of millions of orders) – what to do?

Answer: shard the key. Distribute orders across delay_queue_0 … delay_queue_9 based on a hash of the order ID, and run ten polling threads, increasing throughput tenfold.

Q3: Middleware crashes completely – fallback?

Answer: keep a T+1 offline scan task running on a replica that nightly scans yesterday’s unpaid orders and cancels them. Design with middleware decoupling and eventual consistency in mind.

Interview answer template

Architecture choice : prefer Redis ZSet as a lightweight delayed queue; Score stores expiration timestamp, Value stores order ID.

Core flow : a background thread each second runs ZRANGEBYSCORE to fetch expired orders; use a Lua script for atomic removal and cancellation.

Reliability guarantees : introduce a processing queue with ACK mechanism; ensure the cancel API is idempotent.

Advanced optimization : for massive scale, consider RocketMQ 5.0 arbitrary‑delay messages to offload the service.

Fallback guarantee : retain a low‑frequency database scan task to achieve eventual consistency in extreme cases.

Conclusion

Technical interviews assess not only coding ability but also respect for resources and defense against edge cases. Use Redis for most delayed‑task scenarios, avoid full‑table scans, and combine with MQ or a time wheel when the scale demands it.