Thread Pool Tuning for 10M SMS/Hour: From Golden Formula to Backpressure

Introduction

A developer failed an interview when asked: "We need to send 10 million marketing SMS in 1 hour. How do you design the thread pool? Core parameters? Rejection strategy?" The candidate naively suggested a FixedThreadPool with 500 threads and a large queue. The interviewer immediately exposed three fatal flaws.

Why Executors Is a Production "No-Go Zone"

Big-tech standards forbid Executors.newFixedThreadPool or newCachedThreadPool for two reasons:

OOM Risk: FixedThreadPool uses an unbounded LinkedBlockingQueue (length Integer.MAX_VALUE). Ten million tasks queued before processing will exhaust JVM heap.

Resource Exhaustion: CachedThreadPool allows unlimited thread creation; a sudden burst can drive CPU to 100%.

Conclusion: Production must manually instantiate ThreadPoolExecutor with a bounded queue .

Three Realms of Thread Pool Tuning

Realm 1: Golden Formula for Initial Values

Never guess thread counts. First ask: "Is the task CPU-bound or I/O-bound?" SMS sending involves network calls → typical I/O-bound .

Use the formula: N_cpu: CPU core count U_target: Target CPU utilization (e.g., 0.8) W/C: Wait time / Compute time ratio

Practical landing: For ten-million-scale pushes, W/C is large. Start with 2 * N_cpu * (1 + W/C) (e.g., 200–400) and adjust via load testing.

Realm 2: Dynamic Tuning + Full-Chain Monitoring

Parameters are static; traffic is dynamic. Senior engineers (P7+) adopt dynamic thread pools :

Parameter externalization: CoreSize, MaxSize, QueueSize live in config centers (Apollo, Nacos), not hard-coded.

Monitoring & alerting: Track queue remaining capacity and pool active rate. Trigger alerts or auto-scale when queue exceeds 80% full.

Tip: Mentioning open-source projects Hippo4J or DynamicTp in interviews scores extra points.

Realm 3: Rejection Strategy as the "Ultimate Defense Line"

When the pool saturates, which RejectedExecutionHandler?

AbortPolicy (default): Throws exception → data loss. Never choose.

CallerRunsPolicy (recommended): The submitting thread (e.g., main thread) executes the task itself. This is a natural backpressure mechanism: the producer busy sending SMS cannot fetch new tasks from DB, slowing ingestion and giving the pool breathing room.

Many consider CallerRunsPolicy a pitfall because it blocks the main thread. But in offline batch scenarios (like this SMS job), that "pitfall" becomes a superpower.

Online web scenario (avoid): Occupies Tomcat threads → entire site hangs.

Offline batch scenario (godsend): The "data-fetching thread" sends SMS when the pool is full, automatically throttling production rate and eliminating OOM risk.

Advanced insight: This is backpressure — the producer forced to do consumer work stops producing, giving the pool recovery time.

Final "Anti-Counter" Guide: What If the Service Crashes?

Interviewer: "Tasks sit in memory queue; machine dies; 1 million unsent SMS — how to recover?"

Perfect answer:

Local persistence: Before enqueueing, record "sending" status in DB/Redis.

Ack mechanism: After thread finishes, callback updates status to "completed".

Offline compensation: A scheduled job (T+N) scans tasks stuck in "sending" >10 minutes and re-dispatches them.

Interview Standard Answer Template (Memorize This)

"For 10M SMS push, I avoid Executors shortcuts due to unbounded queue OOM risk.

First, parameter setting: I derive initial values via the golden formula and load-test; being I/O-bound, I start at ~200–400 threads.

Second, rejection strategy: I choose CallerRunsPolicy. Its backpressure lets the main thread help process when overloaded, throttling production speed and keeping the system alive.

Third, dynamic tuning: To handle SMS gateway fluctuations, I integrate a dynamic thread pool framework for real-time queue monitoring and core thread adjustment.

Fourth, reliability: Combined DB status flags and scheduled compensation tasks ensure zero task loss even on machine restart."

Advanced Thinking: Single Machine Survived — Now "Distributed"?

Interviewer's follow-up: "Given a 5-node cluster, design an architecture to send 10M SMS in 1 hour with no duplicates, no omissions, high concurrency ."

Key dimensions to consider:

Task sharding: How do 5 nodes divide work without contention?

State transfer: When a node dies, how do remaining nodes take over its tasks?

Global rate limiting: How to ensure aggregate throughput doesn't overwhelm the vendor gateway?

Single-node tuning is "technique"; cluster architecture is "the Way".

Closing Thoughts

Technical interviews test not memorized parameters but your reverence for system stability . Anticipating OOM, considering backpressure, guaranteeing data reliability — that's what separates you from average developers.