Scaling Spring Boot Sign-In to 100M Users: Distributed Architecture Evolution & Production Hardening

1. Real-World Case: 08:00 Activity Peak

A content app with 100M registered accounts, 20M DAU, and 5M daily sign-ins. Business rules: first sign-in grants 1 point; consecutive days 7, 14, 30 grant extra 10, 30, 100 points. App, H5, and mini-program share the DAILY sign-in capability. Business day uses Asia/Shanghai; clients cannot specify date. Duplicate clicks, gateway retries, and client timeouts must be idempotent. Rewards may be delayed but never lost or duplicated. 45% of users sign in between 08:00–09:00; operational push may concentrate 20% within one minute.

2,250,000 requests / 1 hour ≈ 625 QPS
450,000 requests / 1 minute = 7,500 QPS
Design for 30,000 QPS considering retries and redundancy

Definition of "sign-in success": server has uniquely accepted the account's sign-in command for that business day, and a recoverable persistent command record exists. Points arrival is asynchronous, not part of the HTTP synchronous path. Response example:

{
  "code": 0,
  "data": {
    "signed": true,
    "firstSigned": true,
    "streak": 7,
    "rewardStatus": "PROCESSING",
    "requestId": "01JQ4Z..."
  }
}

Repeated same-day requests still return 200 with firstSigned=false; returning 409 would induce meaningless retries.

2. Define SLOs and Data Authority

SLO Targets and Design Impact

Peak Write : 30,000 QPS — Hot path must not synchronously call points, notification, analytics

API P99 : < 80 ms — Sync ops only: auth, command persistence, Redis atomic state write

Duplicate Requests : No duplicate rewards — Three-layer idempotency: business key, atomic decision, reward ledger

Reward Latency : Seconds to minutes — Kafka async, queryable status and compensation

Single Point Failure : No permanent reward loss — Cannot use Redis Stream as sole Outbox

Historical Audit : Replayable, reconcilable — DB Ledger as source of truth, Redis as rebuildable hot state

Data Responsibility and Authoritative Source

Redis Daily Bitmap : Today sign-in, calendar fast read — Not authoritative — Retention: 400 days, rebuildable

Redis Streak : Fast consecutive days return — Not authoritative — Retention: Active users hot state

Redis Stream : Wake up Dispatcher — Not authoritative — Retention: Capacity and oldest event watermark

signin_command : Accepted command recovery anchor — Authoritative — Retention: Cover compensation window

signin_ledger : Sign-in fact — Authoritative — Retention: Long-term archive

point_ledger : Reward fact — Authoritative — Retention: Long-term archive

This table is the premise: when Redis master-slave switch loses recent writes, rebuildable state can fix; if a successful business command lacks authoritative record, retries cannot recover it.

3. Architecture Evolution: Driven by Bottlenecks, Not Component Stacking

10K users → MySQL unique index
1M users → Redis hot state + MySQL
10M users → Redis Cluster + async events
100M users → Sharded Bitmap + Command Outbox
Platform → Risk control, data warehouse, multi-region, rule center

3.1 10K Users: Start with Database Unique Index

CREATE TABLE signin_ledger (
  id BIGINT NOT NULL AUTO_INCREMENT,
  biz_id VARCHAR(160) NOT NULL,
  account_id BIGINT NOT NULL,
  source VARCHAR(32) NOT NULL,
  sign_date DATE NOT NULL,
  streak INT NOT NULL,
  created_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
  PRIMARY KEY (id),
  UNIQUE KEY uk_biz_id (biz_id),
  KEY idx_account_date (account_id, sign_date)
) ENGINE=InnoDB;

bizId = SIGNIN:DAILY:{accountId}:20260924

. Direct insert; duplicate key means "already signed in today". At this stage, do not introduce distributed locks, Redis Cluster, Kafka, or Seata; the database unique index is the most reliable, auditable atomic boundary.

3.2 1M Users: Redis for Hot Read/Write, Ledger Still in DB

Concurrent vulnerability in check-then-set:

if (!Boolean.TRUE.equals(redisTemplate.opsForValue().getBit(key, offset))) {
  redisTemplate.opsForValue().setBit(key, offset, true);
  rewardService.addPoints(accountId);
}

Two requests may both read false. First decision must come from atomic operation's old value, or converge to Lua.

3.3 100M Users: Solve Massive Small Keys and Hot Slots

"One bitmap per user per month" would create 1.2B keys at 100M users. Bitmap itself is small, but Redis object, dictEntry, SDS, key string, and allocator fragmentation are not. Design must shard by date and user bucket.

4. Date + User Sharded Bitmap

Key format: {signin:DAILY:042}:day:20260924 100M users into 1024 buckets:

Max offset per bucket ≈ 97,657 bits
Single bucket daily bitmap ≈ 12 KB
1024 buckets daily raw bitmap ≈ 12.5 MB
400 days ≈ 5 GB (excluding objects, replication, fragmentation)

4.1 Never Use Snowflake ID as Offset

Snowflake ID can approach 10^18. SETBIT key 1983481238732873728 1 would expand to a disastrously large string. Must assign stable, non-reusable dense index at registration:

CREATE TABLE account_uid_index (
  account_id BIGINT NOT NULL,
  uid_index BIGINT NOT NULL,
  PRIMARY KEY (account_id),
  UNIQUE KEY uk_uid_index (uid_index)
) ENGINE=InnoDB;

Hot path reads uid_index from local Caffeine or Redis cache; fallback to account DB on cache miss. Account deletion/merge must not recycle offset, else new account inherits old sign-in bits.

public final class SignInKeyRouter {
  private static final int SHARD_BITS = 10;
  private static final int SHARD_COUNT = 1 << SHARD_BITS;

  public static int bucket(long uidIndex) {
    return (int) (uidIndex & (SHARD_COUNT - 1));
  }
  public static long offset(long uidIndex) {
    return uidIndex >>> SHARD_BITS;
  }
  public static String tag(SignInSource source, int bucket) {
    return "{signin:" + source.code() + ":" + String.format("%03d", bucket) + "}";
  }
}

source

must be server-side enum (e.g., DAILY, NEW_USER_7D). Cannot concatenate request param directly into Redis key; otherwise attackers can create infinite sources, infinite keys, or use } to interfere with Cluster hash tag.

4.2 Why Shard?

Single daily key forces all SETBIT into one Cluster slot. 1024 buckets spread requests across 1024 slots; but 1024 is not magic:

Single bucket peak = total peak ÷ bucket count × hotspot factor
30,000 ÷ 1,024 × 3 ≈ 88 QPS/bucket

Monitor per-bucket QPS, latency, memory. Bucket count change requires dual-read/dual-write migration; cannot change 1024 to 4096 online directly.

5. Business Day & Consecutive Sign-In: State Machine Prone to Midnight Errors

Date must be server-computed:

private static final ZoneId BUSINESS_ZONE = ZoneId.of("Asia/Shanghai");
LocalDate date = LocalDate.now(BUSINESS_ZONE);

Client-sent date causes tampering, timezone inconsistency, and bypass of make-up sign-in. Normal sign-in only accepts current business day; make-up must be permission-controlled, auditable separate command.

Near midnight, out-of-order arrivals: a 2026-09-24 23:59:59 request delayed by network may arrive after 2026-09-25 00:00:01. Blindly updating streak would roll back today's state to yesterday.

State machine (simplified):

No history → First sign-in / streak=1
date == lastDate → Duplicate
date == lastDate+1 → Consecutive / streak+1
date > lastDate+1 → Reset / streak=1
date < lastDate → Stale / no rollback

Gateway records acceptedAt; server only accepts corresponding current business day; late previous-day requests do not backfill sign-in, only return retryable or current state. This prevents state regression.

6. One Atomic Sign-In in Redis Cluster

Keys participating in one Lua must reside in same slot:

{signin:DAILY:042}:day:20260924
{signin:DAILY:042}:streak
{signin:DAILY:042}:stream

streak

is a per-bucket Hash, field = uidIndex, value = epochDay:streak. Acknowledge capacity: 100M users = 100M fields, actual memory far exceeds value strings. Recommend retaining only last 90 days active users; long-inactive users recalc streak from signin_ledger on next sign-in and backfill. Do not set short TTL on entire Hash; one active account keeps whole bucket alive. signin.lua:

-- KEYS[1] bitmap; KEYS[2] streak hash; KEYS[3] dispatch stream
-- ARGV[1] offset; ARGV[2] uidIndex; ARGV[3] epochDay
-- ARGV[4] bitmapTTL; ARGV[5] streamMaxLen
-- ARGV[6] commandId; ARGV[7] accountId; ARGV[8] source; ARGV[9] signDate
local old = redis.call('SETBIT', KEYS[1], ARGV[1], 1)
local raw = redis.call('HGET', KEYS[2], ARGV[2])
local lastDay, streak = -1, 0
if raw then
  local p = string.find(raw, ':')
  lastDay = tonumber(string.sub(raw, 1, p - 1))
  streak = tonumber(string.sub(raw, p + 1))
end
local today = tonumber(ARGV[3])

if old == 1 or today < lastDay then
  return {0, streak, 'UNCHANGED'}
end
if lastDay == today - 1 then streak = streak + 1 else streak = 1 end
redis.call('HSET', KEYS[2], ARGV[2], tostring(today) .. ':' .. tostring(streak))
redis.call('EXPIRE', KEYS[1], tonumber(ARGV[4]))
redis.call('XADD', KEYS[3], 'MAXLEN', '~', tonumber(ARGV[5]), '*',
  'commandId', ARGV[6], 'accountId', ARGV[7], 'source', ARGV[8],
  'signDate', ARGV[9], 'streak', tostring(streak))
return {1, streak, 'ACCEPTED'}

Script guarantees atomic "first bitmap set + streak change + Stream notification" but does not guarantee persistence after Redis failover. Calling it the sole reliable Outbox is wrong; the Command table in next section is the recovery anchor.

7. Reliable Command Outbox: Acknowledge Cross-Storage Boundaries, Don't Hide Them

CREATE TABLE signin_command (
  id BIGINT NOT NULL AUTO_INCREMENT,
  command_id CHAR(26) NOT NULL,
  biz_id VARCHAR(160) NOT NULL,
  account_id BIGINT NOT NULL,
  source VARCHAR(32) NOT NULL,
  sign_date DATE NOT NULL,
  status VARCHAR(24) NOT NULL,
  streak INT NULL,
  payload JSON NOT NULL,
  next_retry_at DATETIME(3) NULL,
  created_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
  updated_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3)
    ON UPDATE CURRENT_TIMESTAMP(3),
  PRIMARY KEY (id),
  UNIQUE KEY uk_biz_id (biz_id),
  KEY idx_status_retry (status, next_retry_at)
) ENGINE=InnoDB;

Recommended sequence:

Client → SignIn Service → INSERT IGNORE command (NEW)
                    → Redis Lua (SETBIT + streak + XADD) → mark ACCEPTED (streak)
                    → 200 PROCESSING + commandId
                    → async send event → consume → ledger unique key + grant

Failure semantics must be explicit:

Command DB write fails: do not call Redis, fail directly.

Redis fails: return "system busy, retry", NEW command not silently executed by background, avoiding user seeing failure then later success.

Redis succeeds, app crashes before mark ACCEPTED: reconciliation job aligns Bitmap with today's NEW commands and补标.

Kafka/Dispatcher fails: ACCEPTED command remains in DB, scanner can redeliver.

Entire Redis cluster lost: rebuild hot state from Command/Ledger.

This is not zero-cost. 30,000 QPS synchronous Command write must shard by account_id / biz_id, and benchmark index, binlog, replication lag, connection pool. If team unwilling to bear this cost, can use persistent Kafka command topic, but API must return "accepted", not immediate sign-in success.

8. Spring Boot Core Implementation

@Service
@RequiredArgsConstructor
public class SignInService {
  private final SignInCommandRepository commandRepository;
  private final UserIndexService userIndexService;
  private final RedisSignInGateway redisGateway;

  public SignInResult signIn(long accountId, SignInSource source) {
    LocalDate date = LocalDate.now(ZoneId.of("Asia/Shanghai"));
    String bizId = String.format("SIGNIN:%s:%d:%s", source.code(), accountId,
        date.format(DateTimeFormatter.BASIC_ISO_DATE));
    SignInCommand command = commandRepository.createOrGet(
        UlidCreator.getUlid().toString(), bizId, accountId, source, date);

    if (command.isAccepted()) return SignInResult.duplicate(command.streak());

    long uidIndex = userIndexService.getCachedUidIndex(accountId);
    RedisSignInResult accepted = redisGateway.accept(
        uidIndex, source, date, command.commandId());

    if (!accepted.accepted()) {
      // Concurrent winner may have just finished Lua, not yet updated command; short poll then decide retry.
      return commandRepository.awaitAccepted(bizId, Duration.ofMillis(50))
          .map(SignInResult::from)
          .orElseThrow(RetryableSignInException::new);
    }
    commandRepository.markAccepted(command.commandId(), accepted.streak());
    return SignInResult.first(accepted.streak());
  }
}

Redis client timeout can retry limited times: Lua's SETBIT returns old value, retry won't re-accept. But must configure short timeout, backoff, jitter, and total retry cap; cannot amplify Redis 20,000 QPS jitter into 100,000 QPS retry storm.

9. Dispatcher: Stream for Low-Latency Notify, DB Scan Guarantees Final Delivery

Each Stream creates Consumer Group:

XGROUP CREATE {signin:DAILY:042}:stream signin-dispatch $ MKSTREAM
XREADGROUP GROUP signin-dispatch worker-a COUNT 200 BLOCK 1000 
  STREAMS {signin:DAILY:042}:stream >

Correct flow:

Stream batch read → batch load ACCEPTED commands by commandId
→ Kafka async batch send → success callback then XACK
→ XAUTOCLAIM take over crashed worker's Pending
→ DB scan ACCEPTED but undelivered/timed-out commands as final safety net

Do not kafkaTemplate.send().get(3 seconds) per message then ACK. If each waits 5 ms, one worker theoretically only 200 msg/s, at 30,000 QPS backlog inevitable. Kafka uses accountId as key for per-account ordering; partitions, batch size, dispatcher concurrency determined by load test.

spring:
  kafka:
    producer:
      acks: all
      properties:
        enable.idempotence: true
        max.in.flight.requests.per.connection: 5
        delivery.timeout.ms: 120000
        request.timeout.ms: 30000

Kafka send success, crash before XACK causes redelivery. This is correct at-least-once semantics; deduplication must be at ledger unique key, not fantasy that message system "never duplicates".

10. Rewards: One Sign-In Command May Produce Multiple Immutable Reward Facts

Day 7 reward is not "10 points" but "daily 1 point + consecutive 7 days extra 10 points". Event must carry rule version and determined amounts:

{
  "commandId": "01JQ4Z...",
  "bizId": "SIGNIN:DAILY:10086:20260924",
  "accountId": 10086,
  "signDate": "2026-09-24",
  "streak": 7,
  "ruleVersion": "daily-2026-09-v3",
  "awards": [
    {"awardBizId": "SIGNIN_DAILY:DAILY:10086:20260924", "type": "POINT", "amount": 1},
    {"awardBizId": "SIGNIN_STREAK_7:DAILY:10086:20260924", "type": "POINT", "amount": 10}
  ]
}

CREATE TABLE point_ledger (
  id BIGINT NOT NULL AUTO_INCREMENT,
  account_id BIGINT NOT NULL,
  award_biz_id VARCHAR(180) NOT NULL,
  point_delta INT NOT NULL,
  rule_version VARCHAR(64) NOT NULL,
  created_at DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
  PRIMARY KEY (id),
  UNIQUE KEY uk_award_biz_id (award_biz_id),
  KEY idx_account_time (account_id, created_at)
) ENGINE=InnoDB;

@Transactional
public void grant(Award award) {
  int inserted = pointLedgerMapper.insertIgnore(
      award.accountId(), award.awardBizId(), award.amount(), award.ruleVersion());
  if (inserted == 1) {
    userPointMapper.increase(award.accountId(), award.amount());
  }
}

Same message consumed 100 times, balance increases at most once. Sign-in fact also persisted by SignInLedgerConsumer via signin_ledger.uk_biz_id; don't only record points, else cannot distinguish "sign-in success but no reward", "rule version change", "sign-in fact missing".

11. Monthly Calendar, DAU & Query Degradation

User fixed to one bucket. All daily keys in a month share hash tag; Lua can read all GETBIT in one round-trip, avoiding 28–31 RTTs:

-- KEYS = monthly bitmap keys; ARGV[1] = offset
local result = {}
for i = 1, #KEYS do
  result[i] = redis.call('GETBIT', KEYS[i], ARGV[1])
end
return result

Beyond Redis 400-day retention, calendar queries signin_ledger. Do not silently return empty calendar.

DAU async execution:

DAU(date) = Σ BITCOUNT({signin:DAILY:bucket}:day:date)

Aggregation written to ClickHouse/metrics DB; user sign-in hot path does no global stats.

12. Morning Peak Incident Retrospective: Kafka Unavailable 20 Minutes

08:00:00 activity starts, service pre-scaled from 8 to 32 pods. 08:03 Kafka ACL misconfig, all producers fail; Redis and Command DB healthy.

08:03 Kafka unavailable
Dispatcher send fails → no XACK
Stream Pending grows
ACCEPTED commands not marked delivered
Alert: oldest pending > 60s
Alert: command pending age > 60s
08:23 Kafka ACL fixed
XAUTOCLAIM + DB scanner redeliver
Ledger/Reward idempotent write via unique keys
08:31 backlog cleared, reconciliation passed

User sign-in API still returned PROCESSING quickly after Redis Lua success, no synchronous Kafka wait. Post-recovery redelivery is expected; awardBizId unique constraint ensures no duplicate points.

Two operational requirements exposed:

Stream MAXLEN and Redis memory must cover max tolerable Kafka downtime; if exceeds retention watermark, trigger throttling or degradation, because trimming unprocessed Stream loses events.

Even if Stream trimmed, signin_command scanner can still redeliver; hence it is the reliability anchor.

13. Redis Failure & Data Rebuild

Redis is not sole audit source. Rebuild daily bitmap by scanning recent retention window of signin_ledger, recalc bucket and offset via uid_index:

for (SignInLedger row : ledgerRepository.scanSince(cutoffDate)) {
  long uidIndex = userIndexService.getUidIndex(row.accountId());
  int bucket = SignInKeyRouter.bucket(uidIndex);
  redisTemplate.opsForValue().setBit(
      dayKey(row.source(), bucket, row.signDate()),
      SignInKeyRouter.offset(uidIndex), true);
}

Streak can rebuild by scanning each account's recent consecutive ledger dates. Don't let rebuild task run unbounded SETBIT; group by bucket, pipeline batch, rate-limit, monitor Redis CPU, network, latency.

Failure Recovery Matrix

Redis timeout : User gets retryable error → Limited retry; NEW command cleanup

Redis success then app crash : User may timeout → Bitmap vs NEW command reconciliation补标

Kafka unavailable : Sign-in succeeds, reward processing → Stream + DB scanner redeliver

Dispatcher crash : Sign-in succeeds → XAUTOCLAIM reclaim Pending

Reward DB failure : Reward delayed → Kafka retry/DLT, replay by awardBizId Redis total data loss : Recent hot queries degrade → Batch rebuild from command/ledger

If product requires "Redis total loss, all returned-success sign-ins absolutely not lost", cannot rely on Redis Lua as sole acceptance point; must use DB unique key or persistent Kafka command log first. This is business semantics choice, not solvable by increasing Redis replicas.

14. API, Risk Control & Rate Limiting

@PostMapping
public ApiResponse<SignInResponse> signIn(
    @AuthenticationPrincipal LoginUser user,
    @RequestParam(defaultValue = "DAILY") String source) {
  SignInSource checked = SignInSource.parse(source); // whitelist validation
  SignInResult result = signInService.signIn(user.accountId(), checked);
  return ApiResponse.ok(SignInResponse.from(result));
}

Chain: CDN/WAF → Gateway global rate limit → accountId rate limit → Sign-in service → Redis.

Same account: 3 requests per 5 seconds. Rate limiting protects resources, idempotency protects correctness; they cannot substitute each other. If rewards redeemable for cash, coupons, scarce rights, additionally send device, IP, UA, account age async for risk control; high-risk accounts can sign-in succeed but reward status REVIEWING; don't put complex models on hot path.

15. Kubernetes, Timeouts & Capacity Governance

resources:
  requests: { cpu: "500m", memory: "768Mi" }
  limits:   { cpu: "2", memory: "1536Mi" }
startupProbe:
  httpGet: { path: /actuator/health/liveness, port: 8080 }
  failureThreshold: 30
  periodSeconds: 2
readinessProbe:
  httpGet: { path: /actuator/health/readiness, port: 8080 }
  periodSeconds: 5
terminationGracePeriodSeconds: 40

Also need PDB, topologySpreadConstraints, connection pool limits, graceful shutdown. preStop sleep only buffers Endpoint updates, cannot replace app stopping new requests and draining in-flight.

HPA should not only watch CPU: Redis, DB, Kafka waits make CPU low while requests queue. Combine CPU, HTTP in-flight, P99, Redis pool wait, Command backlog. Predictable 08:00 peak pre-scale at 07:50; HPA handles unpredictable spikes.

Layered timeout example:

Gateway: 1000 ms
SignIn API budget: 80 ms
Redis command: 50~100 ms
Kafka: only Dispatcher async, not in API budget

Java 21 virtual threads reduce platform thread pressure from blocking I/O, but do not increase Redis single-shard throughput, DB connections, or Kafka broker capacity.

16. Observability, Reconciliation & Canary

At minimum monitor:

signin_request_total / duplicate_total / error_total / latency
redis_lua_latency / redis_timeout_total / hot_bucket_qps
command_new_total / accepted_total / new_oldest_age_seconds
stream_pending_total / stream_oldest_pending_age_seconds
kafka_consumer_lag / dispatcher_send_failure_total / dlq_total
reward_success_total / reward_duplicate_total / reward_pending_age_seconds
command_vs_ledger_gap_total / expected_award_vs_point_ledger_gap_total

Logs, Command, Kafka Event, Ledger all carry requestId, commandId, bizId, accountId, source, signDate, ruleVersion. When user says "sign-in succeeded but points not arrived", can locate to acceptance, delivery, consumption, ledger within minutes.

Canary rule changes: events carry immutable ruleVersion and awards.amount. Same user routed to different versions, consumer won't recalc historical events with "current rule". Rule amounts, canary audience, effective time audited by rule center; forbidden to rely solely on if (streak == 7) in app code.

17. Load Testing & Pre-Launch Checklist

Load test not just verify HTTP 200. Must include:

30,000 QPS first sign-in sustained 10 min, observe full-chain P99, Redis bucket distribution, DB command write, Kafka backlog.

Same account 100 concurrent, only one first sign-in, one signin_ledger, correct count of point_ledger.

Midnight mixed requests, streak no rollback.

Inject Redis P99 +100 ms, retry capped, no storm.

Kafka unavailable 30 min, verify Stream watermark, Command scanner, recovery redelivery.

Kill Dispatcher, verify XAUTOCLAIM takes over Pending.

Resend same event 100 times, balance increases once.

Flush Redis, rebuild from Ledger/Command, sample compare calendar and streak.

In k6, don't use per-VU reset __ITER as global user ID, else "first sign-in test" incorrectly becomes massive duplicate sign-ins. Use global iteration, real token pool, split metrics for first, duplicate, auth failure, risk reject scenarios.

Pre-launch confirm:

Business day, late request, make-up rules confirmed. uid_index non-reusable, cache miss path load tested.

All Lua keys in same Cluster slot.

Streak Hash real memory load tested, long-inactive governance in place.

Stream trim watermark covers failure window, with alerts.

Dispatcher has consumer group, batch send, async ACK, Pending reclaim, DB scan fallback.

Rewards idempotent via independent awardBizId and DB unique index.

Command, Sign-in Ledger, Points Ledger have auto-reconciliation and compensation.

Pre-scale for peak, downstream Redis/Kafka/MySQL capacity rehearsed.

18. Summary

A releasable, runnable sign-in solution must not stop at "Redis SETBIT + BITCOUNT ". It must clarify boundaries:

First hot state decision: Redis Lua
Business command recovery anchor: Command Outbox
Low-latency delivery: Redis Stream + Dispatcher
Final event transport: Kafka
Sign-in & reward facts: respective Ledgers + unique business keys
Failure recovery: reconciliation, compensation, rebuild Redis from authoritative ledgers
Peak stability: pre-scale, rate limiting, timeout budgets, load test drills

When team can answer "Redis succeeds but app crashes", "Stream trimmed", "Midnight request out-of-order", "Day 7 exactly how many reward entries", "Kafka down 20 min how to reconcile", this sign-in system has truly moved from Demo to Production.