Why Polling Breaks at Scale and How Spring Boot Webhooks Enable High‑Concurrency Event‑Driven Architecture

1. The Problem with Polling

Many early systems use a scheduled @Scheduled(fixedDelay = 3000) task that repeatedly queries the database and external services. This works for low traffic but fails at scale for four reasons:

Structural resource waste: empty queries, hotspot indexes on WAITING rows, and unnecessary third‑party calls.

Latency vs cost trade‑off: a 10‑second interval adds 10 seconds of delay; shortening the interval overloads DB and APIs.

Scaling does not fix the design: adding more instances multiplies table scans and can cause duplicate processing.

Operational complexity shifts to ops (scheduling, lock contention, compensation scripts, instability).

Key insight: Polling treats “infrequent state changes” as “continuous queries”. The correct model is to emit an event only when a state actually changes.

2. Webhook vs. Polling

Data acquisition: consumer pulls vs. producer pushes.

Real‑time: depends on poll interval vs. typically millisecond‑to‑second latency.

Resource consumption: many empty queries vs. only on‑event deliveries.

Scalability: poll‑driven tables can become hot; webhook naturally decouples and scales.

Architectural complexity: low at start for polling, high later; webhook is slightly higher initially but remains stable.

A production‑grade webhook is not a single HTTP endpoint but a full reliable event‑delivery system.

3. When to Adopt Event‑Driven Webhooks

Payment result notifications.

Order status sync (create, ship, cancel, etc.).

Logistics tracking updates.

Open‑platform capability exposure.

AI/workflow callbacks.

Risk‑control or message‑center alerts.

Typical characteristics:

Sparse state changes with massive query volume.

Clear cross‑system notification needs.

Latency‑sensitive business.

Heterogeneous downstream systems.

4. Architecture Blueprint (Six Layers)

Business event production layer: domain services emit domain events.

Transactional outbox layer: same DB transaction writes both business rows and outbox rows.

Message bus layer: Kafka buffers, de‑duplicates, orders events.

Delivery orchestration layer: routing, signing, rate‑limiting, retry, dead‑letter handling.

Subscription governance layer: versioning, tenant isolation, gray‑release.

Observability & ops layer: metrics, alerts, replay, audit.

4.1 Recommended Overall Diagram

Business Service → Outbox Table → Debezium (or scheduled scanner) → Kafka Topic → Webhook Dispatcher → Subscription Matching → HTTP Delivery → Delivery Table (audit) → Retry / DLQ.

5. Core Design Principles (Eight Rules)

At‑least‑once delivery: guarantee no loss, allow duplicates, downstream must be idempotent.

Idempotency over “no duplicate”: use globally unique eventId and per‑delivery deliveryId for deduplication.

Ordering only within the same aggregate root: use aggregateId (e.g., orderId) as Kafka partition key.

Fast failure & async retry: connection timeout 1‑3 s, read timeout 3‑5 s, immediate failure response, retry handled asynchronously.

Separate models for subscription, event, delivery: three tables – webhook_subscription, webhook_event, webhook_delivery.

Transactional outbox is mandatory: ensures event is persisted even if the message broker is temporarily unavailable.

Downstream is untrusted: assume failures, timeouts, malformed responses; isolate each subscriber.

Observability equals reliability: metrics, tracing, replay, and alerting are first‑class citizens.

6. Production‑Ready Spring Boot Code (Key Snippets)

6.1 Domain Objects

package com.example.webhook.domain;

public class DeliveryTask {
    private final String deliveryId;
    private final String eventId;
    private final Long subscriptionId;
    private final String tenantId;
    private final String eventType;
    private final String aggregateId;
    private final String callbackUrl;
    private final String secret;
    private final String payload;
    private final int attemptNo;
    private final LocalDateTime eventTime;
    private final String traceId;
    // constructor generates UUID for deliveryId …
}

6.2 Signature Utilities

package com.example.webhook.support;

public final class SignatureUtils {
    private SignatureUtils() {}
    public static String hmacSha256(String content, String secret) { … }
    public static String sign(String timestamp, String nonce, String payload, String secret) { … }
}

6.3 WebClient Configuration (non‑blocking)

package com.example.webhook.config;

@Configuration
public class WebClientConfig {
    @Bean
    public WebClient webhookWebClient() {
        HttpClient httpClient = HttpClient.create()
            .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 2000)
            .responseTimeout(Duration.ofSeconds(5))
            .doOnConnected(conn -> conn
                .addHandlerLast(new ReadTimeoutHandler(5, TimeUnit.SECONDS))
                .addHandlerLast(new WriteTimeoutHandler(5, TimeUnit.SECONDS)));
        return WebClient.builder()
            .clientConnector(new ReactorClientHttpConnector(httpClient))
            .build();
    }
}

6.4 Bulkhead (per‑subscription isolation)

package com.example.webhook.delivery;

public class SubscriberBulkheadRegistry {
    private final Map<Long, Semaphore> semaphoreMap = new ConcurrentHashMap<>();
    public Semaphore get(Long subscriptionId, int permits) {
        return semaphoreMap.computeIfAbsent(subscriptionId, k -> new Semaphore(permits));
    }
}

6.5 Delivery Executor

package com.example.webhook.delivery;

@Component
@RequiredArgsConstructor
public class WebhookHttpExecutor {
    private final WebClient webhookWebClient;
    private final SubscriberBulkheadRegistry bulkheadRegistry;

    public Mono<DeliveryResult> deliver(DeliveryTask task, int permits) {
        Semaphore semaphore = bulkheadRegistry.get(task.subscriptionId(), permits);
        if (!semaphore.tryAcquire()) {
            return Mono.just(DeliveryResult.reject(task.deliveryId(), "bulkhead full"));
        }
        String timestamp = String.valueOf(Instant.now().toEpochMilli());
        String nonce = UUID.randomUUID().toString();
        String signature = SignatureUtils.sign(timestamp, nonce, task.payload(), task.secret());
        return webhookWebClient.post()
            .uri(task.callbackUrl())
            .contentType(MediaType.APPLICATION_JSON)
            .header(HttpHeaders.USER_AGENT, "webhook-dispatcher/1.0")
            .header("X-Event-Id", task.eventId())
            .header("X-Delivery-Id", task.deliveryId())
            .header("X-Event-Type", task.eventType())
            .header("X-Tenant-Id", task.tenantId())
            .header("X-Aggregate-Id", task.aggregateId())
            .header("X-Attempt-No", String.valueOf(task.attemptNo()))
            .header("X-Timestamp", timestamp)
            .header("X-Nonce", nonce)
            .header("X-Signature", signature)
            .header("X-Trace-Id", task.traceId())
            .bodyValue(task.payload())
            .exchangeToMono(resp -> resp.bodyToMono(String.class).defaultIfEmpty("")
                .map(body -> {
                    int status = resp.statusCode().value();
                    return resp.statusCode().is2xxSuccessful()
                        ? DeliveryResult.success(task.deliveryId(), status, body)
                        : DeliveryResult.failed(task.deliveryId(), status, body);
                }))
            .onErrorResume(ex -> {
                log.warn("Webhook delivery failed, deliveryId={}", task.deliveryId(), ex);
                return Mono.just(DeliveryResult.exception(task.deliveryId(), ex.getMessage()));
            })
            .doFinally(sig -> semaphore.release());
    }
}

6.6 Delivery Result Record

package com.example.webhook.delivery;

public record DeliveryResult(
    String deliveryId,
    boolean success,
    boolean retryable,
    Integer statusCode,
    String responseBody,
    String errorMessage) {
    public static DeliveryResult success(String id, Integer code, String body) { … }
    public static DeliveryResult failed(String id, Integer code, String body) { … }
    public static DeliveryResult exception(String id, String msg) { … }
    public static DeliveryResult reject(String id, String msg) { … }
}

6.7 Kafka Consumer & Dispatcher

package com.example.webhook.consumer;

@Component
@RequiredArgsConstructor
public class WebhookEventConsumer {
    private final SubscriptionRepository subscriptionRepository;
    private final WebhookHttpExecutor webhookHttpExecutor;
    private final DeliveryRecordService deliveryRecordService;
    private final RetryScheduleService retryScheduleService;

    @KafkaListener(topics = "webhook.events", groupId = "webhook-dispatcher")
    public void consume(ConsumerRecord<String, String> record) {
        var message = WebhookEventMessage.from(record.value());
        var subscriptions = subscriptionRepository.findActiveByTenantAndEventType(message.tenantId(), message.eventType());
        Flux.fromIterable(subscriptions)
            .map(sub -> deliveryRecordService.createTask(message, sub))
            .flatMap(task -> webhookHttpExecutor.deliver(task, subscriptionPermits(task))
                .map(res -> new DeliveryBundle(task, res)), 32)
            .doOnNext(this::handleResult)
            .blockLast();
    }
    private int subscriptionPermits(DeliveryTask task) { return 20; }
    private void handleResult(DeliveryBundle bundle) { /* success / retry / dead‑letter */ }
    private record DeliveryBundle(DeliveryTask task, DeliveryResult result) {}
}

6.8 Retry Back‑off Service

package com.example.webhook.service;

@Service
public class RetryBackoffService {
    public LocalDateTime nextRetryTime(int attemptNo) {
        long seconds = switch (attemptNo) {
            case 1 -> 10;
            case 2 -> 30;
            case 3 -> 60;
            case 4 -> 300;
            case 5 -> 900;
            default -> 3600;
        };
        return LocalDateTime.now().plusSeconds(seconds);
    }
    public boolean shouldMoveToDeadLetter(int attemptNo) { return attemptNo >= 6; }
}

6.9 Inbound Receiver (example for a subscriber)

package com.example.receiver.api;

@RestController
@RequiredArgsConstructor
public class OrderWebhookController {
    private final InboundEventService inboundEventService;

    @PostMapping("/webhooks/order")
    public ResponseEntity<String> receive(@RequestBody String body,
                                         @RequestHeader("X-Event-Id") String eventId,
                                         @RequestHeader("X-Timestamp") String timestamp,
                                         @RequestHeader("X-Nonce") String nonce,
                                         @RequestHeader("X-Signature") String signature) {
        String secret = "replace-with-real-secret";
        String localSignature = SignatureUtils.sign(timestamp, nonce, body, secret);
        if (!localSignature.equals(signature)) {
            return ResponseEntity.status(403).body("invalid signature");
        }
        boolean accepted = inboundEventService.accept(eventId, timestamp, nonce, body);
        return accepted ? ResponseEntity.ok("accepted") : ResponseEntity.ok("duplicate");
    }
}

package com.example.receiver.service;

@Service
@RequiredArgsConstructor
public class InboundEventService {
    private final JdbcTemplate jdbcTemplate;

    @Transactional
    public boolean accept(String eventId, String timestamp, String nonce, String body) {
        int inserted = jdbcTemplate.update("""
            INSERT INTO inbound_event(event_id, payload, status, created_at)
            VALUES (?, ?, 'NEW', NOW())
            ON DUPLICATE KEY UPDATE event_id = event_id
            """, eventId, body);
        if (inserted == 0) return false;
        jdbcTemplate.update("""
            INSERT INTO inbound_nonce(nonce, expire_at)
            VALUES (?, DATE_ADD(NOW(), INTERVAL 5 MINUTE))
            """, nonce);
        return true;
    }
}

7. Engineering‑Scale Enhancements

Kafka partition design: topic webhook.events, partition key aggregateId, number of partitions based on peak QPS (e.g., 16‑32).

Thread‑pool isolation: separate pools for Kafka consumption, HTTP delivery, and retry/DLQ handling.

Subscription‑level rate limiting: global, tenant, and per‑subscription limits using Redis token bucket or in‑process Semaphore + RateLimiter.

Slow subscriber degradation: bulkhead‑based temporary circuit‑break, quota reduction, or automatic pause.

Hotspot throttling: split event consumption from fan‑out; generate delivery tasks per subscription and enqueue them separately.

Batch delivery: aggregate events for the same subscriber within 100‑500 ms windows and POST a JSON array.

8. Security Hardening (Beyond HMAC)

Signature includes timestamp\nnonce\npayload and is verified with a shared secret.

Timestamp window ≤ 5 minutes; reject stale requests.

Nonce stored in Redis with short TTL to prevent replay.

Optional network hardening: IP whitelist, mTLS, dedicated VPN, domain‑level certificate management.

9. Observability Design

Essential Prometheus metrics (example names):

webhook_events_consumed_total

webhook_deliveries_total

webhook_deliveries_qps

webhook_delivery_success_total

webhook_delivery_failure_total

webhook_delivery_retry_total

webhook_delivery_dead_total

Latency dimensions: event‑to‑first‑delivery, HTTP call latency, end‑to‑end delivery time.

Queue health: Kafka lag, outbox pending count, retry queue size, dead‑letter count.

Dimensional slicing: tenant, subscription, event type, HTTP status, retry attempt.

10. Dead‑Letter, Compensation & Reconciliation

Management UI supports query by event ID or subscriber, view failure history, inspect last request/response, manual or batch replay.

Business‑level reconciliation (e.g., payment status vs. downstream acknowledgment) to catch false‑positive successes.

Automated compensation jobs (e.g., T+5 min or T+1 h checks) trigger replay or alert when critical downstreams miss events.

11. Real‑World Case Study: Payment Platform

Three evolution stages:

Early polling: every 3 s scans order table, causing massive index pressure and 6‑15 s payment latency.

Direct callback from business thread: improves latency but blocks the payment API when downstream is slow, leading to snow‑ball failures.

Event‑driven webhook platform: payment success writes business row + outbox, Debezium → Kafka, independent dispatcher with per‑tenant limits. Result: API response time ↓ 70 %, peak callback QPS ↑ from 3k to 20k, success rate > 99.95 %, fault‑isolation time reduced from hours to minutes.

12. Common Pitfalls

Embedding callbacks inside the main DB transaction.

Missing delivery table – cannot know which subscriber failed.

Retry logic that sleeps inside the Kafka consumer thread.

Relying solely on DB unique key for idempotency while still bubbling exceptions.

Neglecting downstream contract – no response‑time SLA, no signature verification.

13. Advanced Evolution Paths

Multi‑protocol delivery (Kafka push, RocketMQ, gRPC, email/SMS/IM).

Rule‑based routing (only specific stores, regions, or order amounts).

Versioned event schemas (v1, v2 payloads) with subscriber‑side version selection.

Schema Registry for contract governance, compatibility checks, audit.

14. Team‑Size Recommendations

Small teams: Spring Boot + MySQL + Kafka + outbox scanner + WebClient + delivery + Prometheus/Grafana.

Mid‑large teams: add Debezium, multi‑topic Kafka, dedicated dispatcher cluster, Redis rate‑limit, Resilience4j bulkhead/circuit‑breaker, distributed tracing (SkyWalking/Zipkin), dead‑letter UI, reconciliation service.

Ultra‑reliable scenarios (payments, settlement): strict subscriber onboarding, mTLS, multi‑region active‑active deployment, automated audit & replay, strict SLA monitoring.

15. Final Checklist (10 Steps)

Identify core business events.

Model event, outbox, delivery tables.

Persist events and outbox rows in the same DB transaction.

Stream outbox changes to Kafka (or scheduled scanner).

Use eventId for global idempotency.

Isolate each subscriber with bulkhead, timeout, and rate‑limit.

Implement retry, dead‑letter, and replay mechanisms.

Expose metrics, tracing, audit logs, and reconciliation dashboards.

Enforce subscriber contract (response time ≤ 5 s, signature verification, idempotent handling).

After the core is stable, add batch delivery, rule routing, versioning, and multi‑protocol extensions.

Polling will not disappear overnight, but in a high‑concurrency, low‑latency, well‑governed system it should be retired. The next generation focuses on reliable event emission rather than faster table scans.