Building a Multi-Channel Notification Center: Template Design, Async Processing & Reliable Retry Patterns

Overall Pipeline Architecture

The notification center acts as a pipeline: business services call an API with notification type, business ID, channel list, receiver, and template parameters. The access layer validates, ensures idempotency, creates a NotificationTask persisted with status INIT, then hands off to async execution (thread pool or MQ). Workers pick up tasks, render templates, route to a Channel/Provider, send, and update status to SUCCESS, FAILED, or schedule retry. Persisting every task upfront is the first principle — without a database record, there is no trace when users claim non-delivery.

Template Rendering with Thymeleaf

Instead of String.format or hard-coded HTML, each notifyType maps to a template group with per-channel variants (e.g., ORDER_TIMEOUT_EMAIL HTML, ORDER_TIMEOUT_SMS text). Templates live in the database (or Git) so content changes need no redeploy. A thin TemplateRenderService uses StringTemplateResolver to render stored template strings with a Map<String, Object> of parameters, supporting both TemplateMode.HTML and TemplateMode.TEXT. Rendering is a pure function — given the same template and params, output is reproducible, simplifying debugging and testing.

@Component
public class TemplateRenderService {
    private final ThymeleafTemplateEngine engine;
    public String render(String templateContent, Map<String, Object> params, TemplateMode mode) {
        StringTemplateResolver resolver = new StringTemplateResolver();
        resolver.setTemplateMode(mode);
        engine.setTemplateResolver(resolver);
        Context context = new Context();
        context.setVariables(params);
        return engine.process(templateContent, context);
    }
}

Email templates use full HTML with th:text and th:href; SMS templates are plain text with ${var} placeholders. Channel-specific length limits (e.g., 70-char SMS) are handled inside each template.

Core Abstractions: Channel and Provider

Two interfaces decouple how a message reaches the user ( Channel) from who actually sends it ( Provider). Channel — represents a delivery medium (EMAIL, SMS, STATION). Methods: type(), support(Notification) (pre-condition checks like “user has phone number”), send(Notification). Provider — concrete vendor adapter (e.g., AliyunSmsProvider, TencentSmsProvider). For SMS: name(), send(SmsRequest) returning SmsSendResult with vendor requestId. EmailChannel wraps Spring’s JavaMailSender to send MIME HTML. SmsChannel holds a list of SmsProvider s and delegates to an SmsRouter that selects one based on rules (primary/backup, health, cost). StationChannel simply inserts a row into an in-app message table. All channels are collected into a Map<ChannelType, Channel> via ChannelRegistry for fast lookup.

public interface Channel {
    ChannelType type();
    boolean support(Notification notification);
    SendResult send(Notification notification);
}

public interface SmsProvider {
    String name();
    SmsSendResult send(SmsRequest request);
}

Async Execution: Thread Pool → Message Queue

Synchronous calls block business threads on slow external APIs. For monoliths, a dedicated ThreadPoolTaskExecutor (core 4, max 16, queue 10000, CallerRunsPolicy) suffices. The API saves the task and submits a Runnable to the pool. When the system evolves to microservices, the notification center becomes a separate service: the API publishes the task to RocketMQ/RabbitMQ, and a consumer service processes it. Crucially, the database remains the source of truth — MQ is only a trigger. If a consumer crashes after pulling a message, the task stays INIT in DB; a scheduled retry job re-queues it. This dual mechanism (MQ for speed, DB for durability) prevents loss.

@Bean("notificationExecutor")
public ThreadPoolTaskExecutor notificationExecutor() {
    ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
    executor.setCorePoolSize(4);
    executor.setMaxPoolSize(16);
    executor.setQueueCapacity(10000);
    executor.setThreadNamePrefix("notify-exec-");
    executor.setRejectedExecutionHandler(new CallerRunsPolicy());
    executor.initialize();
    return executor;
}

Rate Limiting & Backpressure

Guava RateLimiter protects downstream vendor QPS limits (e.g., 50 SMS/sec). On tryAcquire failure, the task throws a RetryableException, status reverts to WAITING with nextRetryTime = now + 1s, and the retry job picks it up next cycle — no alert spam for expected backpressure.

Reliability: Persistent State Machine & Retry

The NotificationTask table captures the full lifecycle:

Keys: id, biz_id + biz_type (idempotency), channel_type, notify_type, template_code Payload: receiver (encrypted), params_digest (encrypted), content (rendered full text)

State: status ∈ { INIT, SENDING, SUCCESS, FAILED, DEAD}

Retry: retry_count, max_retry_count, next_retry_time, last_error Flow: save(INIT) → async → SENDING → channel.send() → success → SUCCESS; failure → check retryable → if yes, retry_count++, compute exponential backoff (1m, 5m, 30m…), set next_retry_time, revert to INIT / PENDING; if no or max retries reached → FAILED → eventually DEAD. A @Scheduled job (every 30s) scans for tasks with next_retry_time <= now and retry_count < max_retry_count, re-submits them. For SMS timeouts where vendor state is unknown, the saved vendor_req_id enables a status-query call before blind retry to avoid duplicate delivery.

@Scheduled(fixedDelay = 30000)
public void retryFailedTask() {
    List<NotificationTask> tasks = taskMapper.selectRetryable(LocalDateTime.now(), 100);
    for (NotificationTask task : tasks) {
        if (task.getRetryCount() >= task.getMaxRetryCount()) {
            handleDead(task);
            continue;
        }
        task.setStatus(TaskStatus.SENDING);
        taskMapper.updateById(task);
        sendProcess(task.getId());
    }
}

Dead tasks trigger a DingTalk alert to ops with masked receiver and error details. Verification-code dead tasks allow immediate re-creation (old code invalidated); marketing dead tasks require manual review.

Routing, Fallback & Rate Limiting

SmsRouter

maintains in-memory provider health (updated by a scheduled mock-send or circuit-breaker check). On send, it picks the first healthy provider; if primary fails (errors or high timeout ratio), it fails over to backup. Cross-channel fallback (e.g., SMS → email) is not automatic — the request carries an allowChannels set; only listed channels are tried. Rate limiting uses Redis counters per (phone, notifyType) with TTL; on limit hit, the service can either reject, delay-retry, or downgrade to a lower-intrusion channel (e.g., marketing SMS → in-app message).

public boolean canSend(String phone, NotifyType type) {
    String key = "notify:limit:" + type.getCode() + ":" + phone;
    Long count = redisTemplate.opsForValue().increment(key);
    if (count == 1) redisTemplate.expire(key, Duration.ofMinutes(1));
    return count <= maxPermits(type, phone);
}

Idempotency, Data Masking & Audit

Idempotency key = ( biz_type, biz_id, channel_type) enforced by a unique DB index; high-concurrency paths add a Redis SETNX lock (10s TTL) before insert. Sensitive fields (phone, email) are encrypted at rest; logs always use DesensUtil.maskMobile ( 138****1234) and maskEmail ( t***@example.com). An @Around AOP aspect on NotificationService methods serializes desensitized args, records latency, result/error, chosen channel/provider, and vendor response into an async audit log — enabling full traceability for “user didn’t receive” investigations.

@Aspect @Component
public class AuditLogAspect {
    @Around("@annotation(audit)")
    public Object around(ProceedingJoinPoint pjp, Audit audit) throws Throwable {
        String req = DesensUtil.serialize(pjp.getArgs());
        long begin = System.currentTimeMillis();
        try {
            Object result = pjp.proceed();
            auditLogService.syncLog(req, result.toString(), 0);
            return result;
        } catch (Throwable e) {
            auditLogService.syncLog(req, e.getMessage(), 1);
            throw e;
        }
    }
}

Key Takeaways

Start simple but get four foundations right: (1) persist every send attempt, (2) replace string concatenation with a template engine, (3) define clear Channel/Provider contracts, (4) add async + retry. Evolve to MQ, smart routing, and observability only after these are solid. The goal is not technology showcase but guaranteeing message reliability and user experience — a silent lost notification can kill a transaction.