Reliable Delivery for Billion-Scale LongConn Gateways: Tackling ACK Storms

1. Business background and problem statement

In a real‑time messaging platform the system must deliver messages to online users reliably and compensate offline users. Requirements include ten‑million concurrent users, a peak downstream of one million messages per second, mandatory application‑level ACK, offline compensation, horizontal scaling, graceful node restarts, and handling of Kafka backlog and ACK storms. channel.writeAndFlush() only guarantees that bytes entered the kernel; it does not guarantee that the client has read, processed, or persisted the message, so TCP reliability is insufficient for application‑level guarantees.

2. Real‑world case: ACK storm and retry avalanche

A typical IM system wrote messages directly to the client after consuming from Kafka and kept an in‑memory pending‑ACK table. After a version rollout three problems appeared:

Client reconnections caused a surge of offline re‑pushes, amplifying ACK traffic.

Fixed‑interval retries piled up messages, creating a retry avalanche.

Node restarts cleared the in‑memory pending state, leading to duplicate deliveries and difficult reconciliation.

The root cause was treating the ACK path as an after‑thought instead of a first‑class subsystem.

3. Delivery boundaries

The server must know whether a message has been ACKed by the client.

If the client does not ACK, the server may retry a limited number of times.

Offline users must receive compensation.

The whole process should be traceable, auditable, and alarmable.

The design deliberately targets “at‑least‑once” delivery with idempotent consumption rather than strict global exactly‑once.

4. Architecture overview

The architecture separates four concerns: delivery, ACK handling, offline compensation, and routing. Core components are:

Netty Gateway : maintains long connections, encodes/decodes the protocol, writes messages, receives ACKs.

Redis : stores session routing, deduplication keys, pending metadata, and short‑term state renewal.

Kafka push‑topic : carries business messages downstream.

Kafka retry‑topic : carries delayed retries or asynchronous re‑delivery.

Kafka ack‑event : records ACK audit, reconciliation, and async state updates.

Offline Store (Redis Stream, DB, or object storage): persists messages for offline users.

4.1 Component responsibilities

Netty Gateway

: long‑connection maintenance, protocol codec, message write, ACK receive. Redis: session routing, idempotent dedup, pending metadata, short‑term state renewal. Kafka push‑topic: business downstream command/message bus. Kafka retry‑topic: delayed retry or async re‑delivery chain. Kafka ack‑event: ACK audit, reconciliation, async state update.

5. Technical foundations

5.1 Why ACK must be application‑level

An ACK must answer three questions:

Which message is confirmed?

Who confirmed it?

Is the confirmation “received” or “processed”?

Typical ACK payload includes messageId, conversationId (or bizKey), clientSeq, ackType, and ackTime.

5.2 Why retries cannot block the EventLoop

Blocking the I/O thread with sleep and immediate resend leads to EventLoop blockage, memory pressure, and spikes during network jitter. The solution is to separate the real‑time path from the retry path using an independent scheduler or time‑wheel.

5.3 Why Redis List alone is insufficient for offline compensation

Redis List cannot be the sole source of truth because long offline periods create huge lists, removing middle elements is costly, and compensation may be interrupted. The recommended layered storage is:

Short‑term: Redis Stream or List.

Mid‑/Long‑term: Database, object storage, or log storage.

Real‑time pending: In‑memory + Redis metadata.

6. State machine design

The minimal state set is:

CREATED → ROUTED → SENT → ACKED → TIMEOUT → RETRYING → OFFLINE_BUFFERED → RESEND_ON_LOGIN → DEAD_LETTER

Each state is represented by a DeliveryStatus enum and persisted in Redis hashes, for example:

session:{userId} → hash(nodeId, channelId, sessionId, lastSeen, version)
pending:{requestId} → hash(messageId, userId, status, retryCount, expireAt)
dedup:ack:{userId}:{messageId} → string(1) + TTL
offline:{userId} → stream/list
node:sessions:{nodeId} → set(sessionId)

7. End‑to‑end flow

7.1 Online delivery

Client → Redis → Netty Gateway → Kafka Business Service → push‑topic(messageId, userId)
Gateway consumes → looks up session:{userId} → writes to channel
Adds entry to pending:{requestId}
Client ACK → AckHandler confirms → removes pending entry → records audit

7.2 Offline compensation

Client offline → offline:{userId} stores message
User logs in → OfflineCompensator fetches batch → PushService re‑pushes
Successful push marks message as inflight; failures break the loop

7.3 Retry handling

PendingAckManager scans pendingMap every second
If expireAt < now → timeout
  • If retryCount >= maxRetry → move to dead‑letter
  • If channel inactive → store to offline store
  • Else → send to retry‑topic with exponential back‑off

8. Production‑grade code skeleton

8.1 Package layout

com.example.gateway
├── bootstrap
├── protocol
│   ├── codec
│   ├── model
│   └── handler
├── session
├── delivery
│   ├── pending
│   ├── retry
│   ├── offline
│   └── consumer
├── store
├── metrics
└── config

8.2 Protocol models

@Value @Builder
public class PushFrame {
  long requestId;
  String messageId;
  String userId;
  byte type;
  byte[] payload;
  long serverTime;
}

@Value @Builder
public class AckFrame {
  long requestId;
  String messageId;
  String userId;
  long clientTime;
  byte ackType;
}

8.3 Netty pipeline

pipeline.addLast(new IdleStateHandler(readerIdleSec, 0, 0, TimeUnit.SECONDS));
pipeline.addLast(new AckFrameDecoder(maxFrameLength));
pipeline.addLast(new PushFrameEncoder());
pipeline.addLast(connectionHandler);
pipeline.addLast(heartbeatHandler);
pipeline.addLast(ackHandler);

Key points:

Connection lifecycle handling and ACK handling are separated.

Read‑idle detection prevents zombie connections.

Write‑buffer high/low watermarks provide back‑pressure.

8.4 Pending ACK model

@Value @Builder(toBuilder = true)
public class PendingMessage {
  String messageId;
  long requestId;
  String userId;
  String sessionId;
  int retryCount;
  long expireAt;
  DeliveryStatus status;
}

public enum DeliveryStatus { CREATED, SENT, ACKED, TIMEOUT, OFFLINE_BUFFERED, DEAD_LETTER }

8.5 PushService (delivery & registration)

public DeliveryResult push(DeliveryCommand cmd) {
  Channel ch = localSessionRegistry.getChannel(cmd.userId()).orElse(null);
  if (ch == null || !ch.isActive()) {
    offlineMessageStore.save(cmd);
    return DeliveryResult.offline(cmd.messageId());
  }
  PushFrame frame = PushFrame.builder()
      .requestId(cmd.requestId())
      .messageId(cmd.messageId())
      .userId(cmd.userId())
      .type(cmd.type())
      .payload(cmd.payload())
      .serverTime(System.currentTimeMillis())
      .build();
  ch.writeAndFlush(frame).addListener(f -> {
    if (f.isSuccess()) {
      pendingAckManager.add(cmd, ch);
    } else {
      log.warn("push failed, fallback to offline, messageId={}, userId={}", cmd.messageId(), cmd.userId(), f.cause());
      offlineMessageStore.save(cmd);
    }
  });
  return DeliveryResult.sent(cmd.messageId(), cmd.requestId());
}

Important details:

Write to the channel first, then register the pending entry to avoid “sent but not recorded”.

Write failures fall back to the offline store without blocking the I/O thread.

8.6 AckHandler (fast path)

protected void channelRead0(ChannelHandlerContext ctx, AckFrame ack) {
  boolean confirmed = pendingAckManager.confirm(ack);
  if (confirmed) {
    gatewayMetrics.recordAckSuccess();
    return;
  }
  gatewayMetrics.recordDuplicateOrLateAck();
  log.debug("ignore duplicate or late ack, requestId={}, messageId={}", ack.getRequestId(), ack.getMessageId());
}

The handler prioritises in‑memory lookup, discards duplicates quickly, and defers audit to an asynchronous path.

8.7 PendingAckManager (timeout, retry, dedup)

private final Map<Long, PendingContext> pendingMap = new ConcurrentHashMap<>();
private final ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor(r -> new Thread(r, "ack-timeout-scanner"));

public void start() { scheduler.scheduleAtFixedRate(this::scanTimeout, 1, 1, TimeUnit.SECONDS); }

public void add(DeliveryCommand cmd, Channel ch) {
  PendingMessage pending = PendingMessage.builder()
      .messageId(cmd.messageId())
      .requestId(cmd.requestId())
      .userId(cmd.userId())
      .sessionId(cmd.sessionId())
      .retryCount(cmd.retryCount())
      .expireAt(System.currentTimeMillis() + ackTimeoutMillis)
      .status(DeliveryStatus.SENT)
      .build();
  pendingMap.put(cmd.requestId(), new PendingContext(pending, ch, cmd));
}

public boolean confirm(AckFrame ack) {
  return pendingMap.remove(ack.getRequestId()) != null;
}

private void scanTimeout() {
  long now = System.currentTimeMillis();
  pendingMap.forEach((reqId, ctx) -> {
    if (ctx.pending().getExpireAt() > now) return;
    if (!pendingMap.remove(reqId, ctx)) return;
    DeliveryCommand cmd = ctx.command();
    if (cmd.retryCount() >= maxRetry) {
      offlineMessageStore.deadLetter(cmd, "ack timeout over max retry");
      return;
    }
    if (ctx.channel() == null || !ctx.channel().isActive()) {
      offlineMessageStore.save(cmd.nextRetry());
      return;
    }
    retryProducer.send(cmd.nextRetry(), nextDelay(cmd.retryCount() + 1));
  });
}

private Duration nextDelay(int retryCount) {
  long base = 1000L;
  long max = 30000L;
  long delay = Math.min(base * (1L << Math.min(retryCount, 10)), max);
  long jitter = ThreadLocalRandom.current().nextLong(200, 800);
  return Duration.ofMillis(delay + jitter);
}

Key production insights:

ACK timeout scanning runs in a dedicated thread, never blocking the EventLoop.

Timeouts trigger a unified retry chain instead of immediate re‑send.

If the channel is dead, the message is moved to the offline store or dead‑letter queue.

8.8 OfflineCompensator

public void onUserLogin(String userId) {
  List<DeliveryCommand> cmds = offlineMessageStore.fetch(userId, 100);
  for (DeliveryCommand cmd : cmds) {
    DeliveryResult result = pushService.push(cmd);
    if (result.isSent()) {
      offlineMessageStore.markInflight(userId, cmd.messageId());
    } else {
      break; // stop on first failure to avoid overwhelming the user
    }
  }
}

Typical pitfalls:

A single bulk pull can flood a newly logged‑in user; batch size and per‑user concurrency limits are essential.

Mixing offline compensation with real‑time messages can break ordering; compensate per user in a serial queue.

9. Mitigating ACK storms

Keep ACK packets minimal (only essential fields).

Deduplicate ACKs locally with short‑TTL keys (e.g., dedup:ack:{userId}:{messageId}).

Persist ACK audit asynchronously.

Optionally merge ACKs per window (single, range, or up‑to‑max).

Throttle offline compensation per user and per node.

Local fast‑path ACK handling before remote audit.

10. Scaling and resource isolation

Separate layers:

I/O EventLoop (Netty) handles socket reads/writes.

Async worker pool handles serialization and light computation.

Dedicated retry scheduler scans pending ACKs.

Kafka consumer threads are isolated from the EventLoop.

Per‑user processing is serial; different users are processed in parallel (Kafka partitioned by userId and bounded queues inside the gateway).

11. Consistency, idempotency, and failure recovery

Clients must store processed messageId to guarantee idempotent consumption.

Node restarts lose in‑memory pending; minimal pending metadata is persisted in Redis/Kafka.

On restart, unfinished messages are re‑routed to the retry‑topic.

Kafka backlog recovery is rate‑limited and prioritized.

Dead‑letter handling includes manual inspection, automatic re‑push, fallback channels, or business degradation.

12. Observability

Essential metrics:

Online connections, messages/sec, ACK/sec.

ACK latency (average & P99).

Pending map size.

Timeout retries.

Offline compensation rate.

Dead‑letter count.

Kafka backlog size.

Log every ACK with messageId, requestId, userId, sessionId, nodeId, retryCount, and deliveryStatus. Define alerts for rising ACK timeout rate, pending map spikes, offline queue growth, dead‑letter surges, and tenant‑specific ACK QPS anomalies.

13. Configuration, load testing, and operational guidance

Configurable parameters: ACK timeout, max retries, back‑off strategy, offline batch size, per‑user compensation concurrency, Kafka topic names, write‑buffer watermarks.

Load‑test not only message throughput but also ACK peaks, offline recovery, node drain/restart, and Kafka backlog replay.

Run fault‑injection drills: node crash, Kafka partition loss, Redis latency spike, client ACK delay, massive offline user login.

14. Common pitfalls

Assuming TCP guarantees delivery.

Treating all messages with the same high‑cost reliability path.

Compensating offline users too aggressively, causing new ACK storms.

Relying solely on in‑memory pending state.

15. Scope and evolution

The solution fits IM, notification platforms, device‑control commands, and any gateway that needs ACK, retry, and offline compensation. It is not needed for fire‑and‑forget traffic. Future directions include ACK aggregation protocols, multi‑level priority queues, edge‑node local compensation, tenant‑aware rate limiting, and more sophisticated timing wheels.

16. Conclusion

Reliable delivery for long‑connection gateways is achieved by treating ACK, timeout‑driven retry, offline compensation, and node recovery as a closed loop. The design combines explicit state machines, layered storage, back‑pressure, observability, and resource isolation to survive ACK storms, retry avalanches, and large‑scale failures.