10 System Design Questions That Decide Big Tech Interview Success
This article dissects 10 high-frequency system design questions — seckill, short URL, distributed ID, delayed tasks, leaderboard, real-time push, short-video feed, sharding, rate limiting, and config center — with architecture diagrams, core code, and comparative solutions for each.
How to Design a High-Concurrency Seckill System
Core challenges: instant traffic spikes (100-1000x), overselling due to concurrent read/write, and malicious requests. The architecture follows layered rate limiting + async peak shaving + eventual consistency .
Layered Rate Limiting
Frontend: button disabling, CAPTCHA/slider verification, random delay, static HTML on CDN, dynamic seckill URL /getSeckillPath generated at start time.
Gateway (Nginx): limit_req_zone per IP, IP blacklist, token bucket limiting total QPS (e.g., 5000 req/s).
Service: distributed rate limiting via Redis+Lua token bucket, circuit breaker, service isolation.
Cache: pre-warm stock to Redis, use Redis Cluster for sharding.
Async Peak Shaving: after Redis stock deduction, send message to RocketMQ/Kafka; order service consumes asynchronously; frontend polls/WebSocket for result.
Overselling Solution: Redis+Lua Atomic Script
Stock pre-loaded in Redis. Lua script executes atomically (Redis single-threaded): check user not already purchased, check stock > 0, decrement stock, add user to purchased set. Returns 1 (success), 0 (out of stock), -1 (duplicate).
-- Atomic stock deduction script
local stockKey = KEYS[1]
local userKey = KEYS[2]
local userId = ARGV[1]
if redis.call('sismember', userKey, userId) == 1 then return -1 end
local stock = tonumber(redis.call('get', stockKey))
if stock <= 0 then return 0 end
redis.call('decr', stockKey)
redis.call('sadd', userKey, userId)
return 1Why it prevents overselling: Lua script runs as a single atomic operation; no race condition between read and write.
Async Order Creation & Eventual Consistency
On Redis success, immediately return "success, creating order"; push order info to MQ. Order service consumes, creates order, deducts DB stock with optimistic lock:
UPDATE seckill_product SET stock = stock - 1 WHERE id = #{productId} AND stock >= 1If DB update affects 0 rows, stock exhausted (possible Redis over-deduction due to replication lag); compensate by adding stock back to Redis and marking order failed. Dead-letter queue handles order creation failures; scheduled job cancels unpaid orders and replenishes stock.
Hotspot Handling
Business isolation: seckill service deployed separately.
JVM local cache: for extreme hotspots, pre-load portion of stock into local AtomicInteger, spin-deduct, async sync to Redis.
Summary: "Block, Shave, Async"
Block: 99.9% traffic stopped at frontend, gateway, cache.
Shave: MQ turns burst into stream.
Async: non-core flows async.
Short URL Service (TinyURL) Design
Requirements & Estimates
Functional: long→short, short→long (302 redirect), expiry, optional custom alias, stats. Non-functional: high availability, low latency, uniqueness, abuse prevention. Estimate: 1M DAU, 2 short URLs/user/day, 10 visits/short URL → 2M writes/day, 20M reads/day (10:1 read-heavy), ~73GB/year storage.
Core Algorithm: Distributed ID + Base62
Reject hash (MD5/SHA1) due to collisions. Use Snowflake/Leaf for 64-bit globally unique, roughly increasing ID → Base62 encode (0-9a-zA-Z). 6 chars ≈ 56.8B, 7 chars ≈ 3.5T. Example: ID 123456 → w7E.
Architecture & Flow
API Gateway → Short URL Service → ID Generator (Snowflake/Leaf) → Base62 encode → store mapping in MySQL (sharded by short_code prefix) → Redis cache shortCode→longUrl → return short URL. Redirect: lookup Redis, fallback to DB, 302 redirect (not 301, to allow stats).
Key Optimizations
DB: unique index on short_code, index on expire_time, partition by short_code prefix (256 shards).
Cache: Redis with TTL, Bloom filter for cache penetration, mutex for cache breakdown, random TTL for avalanche.
HA: stateless service cluster, DB master-slave, Redis Cluster/Sentinel, rate limiting, circuit breaker, degrade stats under load.
Security: URL blacklist, IP/user rate limiting, short code obfuscation (XOR random seed or hash+salt) to prevent enumeration.
Distributed ID Generator Design
Requirements
Global uniqueness, trend increasing (for DB index), high availability, low latency, info security, scalability, 64-bit length.
Solution Comparison
DB Auto-increment: Simple but single point, low perf; for low concurrency.
UUID: No coordination but long, not increasing, index unfriendly; for non-critical.
Snowflake: High perf, trend increasing, 64-bit; clock dependency; for high concurrency ✅.
Segment (Leaf): Extreme perf, no clock issue; trend increasing only, needs DB; for extreme QPS.
Snowflake Deep Dive
64-bit: 1 sign bit, 41-bit timestamp (69 years), 10-bit machine ID (1024 nodes), 12-bit sequence (4096/ms). Machine ID auto-registered via ZK ephemeral sequential nodes or DB optimistic lock. Clock rollback handling: short rollback (<5ms) spin wait; medium (<1s) switch to backup workerId; long (>1s) throw exception + alert. Ultimate fix: periodic timestamp persistence (Leaf).
// Core generation (synchronized)
long timestamp = timeGen();
if (timestamp < lastTimestamp) throw new RuntimeException("Clock rollback");
if (lastTimestamp == timestamp) {
sequence = (sequence + 1) & sequenceMask;
if (sequence == 0) timestamp = tilNextMillis(lastTimestamp);
} else sequence = 0;
lastTimestamp = timestamp;
return ((timestamp - twepoch) << timestampLeftShift) |
(datacenterId << datacenterIdShift) |
(workerId << workerIdShift) |
sequence;Segment Mode (Leaf)
Proxy fetches ID segments (e.g., 1000 IDs) from DB into dual in-memory buffers; clients RPC-get from buffer (lock-free, millions QPS). Seamless buffer switch. Trade-off: trend increasing only, requires DB for persistence.
Production Hardening
Clock rollback: tiered handling, time sync service, disable auto NTP.
WorkerId allocation: ZK/etcd or DB+heartbeat.
Performance: replace synchronized with CAS, pre-generate ID batches, adjust bit allocation.
Delayed Task System (Order Auto-Cancel)
Requirements
Cancel order + release stock if unpaid after 30 min. Minute-level precision, high availability, million orders/day, idempotent, observable.
Scheme Comparison
DB Polling: Simple but DB pressure, high latency; low traffic.
JDK DelayQueue: Precise but in-memory, not distributed, OOM risk; single node.
Redis ZSet: Good perf, distributed, simple; polling delay, Redis pressure at scale; medium ✅.
RabbitMQ Dead Letter: Reliable, no polling; queue-level TTL imprecise, no dynamic delay change; fixed delay.
Time Wheel: O(1) insert/delete, massive scale; complex, HA hard; high concurrency.
Redis ZSet Solution (Primary)
On order create: ZADD delay:order:cancel {now+30min} orderId (atomic with DB via local transaction). Scheduler (multiple instances, distributed lock) polls per minute: ZRANGEBYSCORE key 0 now LIMIT 100. Lua script atomically fetches and removes tasks. Tasks sent to MQ; consumer checks order status (idempotent), updates to CANCELED, releases stock, removes from ZSet. On payment: ZREM cancels task instantly (O(1)).
Critical Fixes
Task loss: two-phase — mark score to now+5min (pending), after MQ send success, delete; if scheduler crashes, task reappears after 5min.
Idempotency: DB state machine (only WAIT_PAY→CANCELED), MQ message ID dedup, idempotent stock release API.
Backlog: shard ZSet by orderId hash, multiple schedulers per shard, scale consumers.
Dynamic delay: ZADD with new score.
Time Wheel Upgrade
For 10M+ orders/day: Netty HashedWheelTimer in distributed Delay-Server, persistence to Redis/RocksDB, consistent hashing sharding, node failure takeover from persistent store. Used by Ele.me Hermes.
Leaderboard System Design
Core Data Structure: Redis ZSet (Skip List)
O(logN) for add, rank, topN. Score encoding for tie-breaking: score = actualScore * 10^13 - timestamp (earlier ranks higher).
Evolution
Small: MySQL ORDER BY score LIMIT 100 + Redis cache TopN.
Medium: Single Redis ZSet per leaderboard.
Large (100M+ users): Shard by userId hash (e.g., 100 buckets). Each bucket has ZSet. Global TopN: fetch TopN from each bucket, merge sort. Write path async via Kafka → consumer batch updates shards.
Advanced Optimizations
Multi-dimensional: separate ZSet per dimension (score, kills) or composite score.
Periodic leaderboards: daily ZSet rank:day:2026-05-29; cron RENAME / DUMP to cold storage (MySQL/HBase) for history.
Million-user: tiered refresh (top 1000 real-time, rest hourly), local cache (Caffeine) for hot Top1000, cursor-based pagination (timestamp cursor).
HA: Redis AOF+RDB, Cluster/Sentinel, idempotent ZADD overwrites, daily full reconciliation from DB.
Real-Time Message Push System
Architecture
Client ↔ WebSocket → Push Gateway (stateless, Netty) → Route Service (Redis: userId→gateway) → MQ (Kafka) → Push Service → online: direct push via gateway; offline: store in Redis List (sharded by userId) + async persist to MySQL/HBase. ACK mechanism: client acks, gateway retries 3x, then offline store. Gateway restart reloads unacked messages from Redis.
Connection Management
Netty + custom protobuf frame (msgId, ack, type). Heartbeat 30s Ping/Pong, 3 misses → disconnect. Session map in memory (userId→connectionIds). State machine: INIT→AUTH→ONLINE→OFFLINE. Reconnect with exponential backoff, pull missed messages from offline queue.
Routing
Two-layer: gateway registers SET push:route:uid gateway-ip:port EX 120 on auth (heartbeat renews). Push service queries Redis by uid, forwards via internal gRPC/Netty to target gateway. Optimization: consistent hashing to fix gateway per user, avoid Redis hot key.
Reliability & Scale
Message ID (Snowflake) for dedup/ordering; same user → same Kafka partition.
Broadcast: batched, rate-limited, separate broadcast topic.
HA: stateless gateways, rolling deploy, K8s HPA, multi-region.
Short Video Feed Flow (10M DAU)
Push-Pull Hybrid
Normal users (fans < 100k): push model — on publish, async write videoId+timestamp to each fan's feed:{uid} ZSet (inbox). Big Vs (fans > 100k): pull model — publish writes only to author's outbox:{v_uid} ZSet. On refresh: merge inbox ZSet (recent N) + real-time pull from followed big V outboxes (multi-way merge in memory). Big V outbox cached locally for few seconds.
Storage
Core metadata: MySQL (sharded) — strong consistency, transactions.
Feed index: Redis ZSet — sorted, high perf.
History: HBase — PB scale, random RW.
Video files: OSS + CDN — unlimited, edge acceleration.
Search: Elasticsearch — full-text.
Performance
Multi-level cache: client (preload 3), CDN (video), Redis (feed index, video meta), local (hot big V).
Full async: Kafka decouples publish, transcode, feed generation, notification.
Cold/hot separation: 7 days Redis, 7d-3m HBase, 3m+ archive.
Rate limiting, circuit breaker, degrade non-core (comments) under load.
Sharding (10M+ Rows)
Strategy
Vertical split first (business domains), then horizontal. Sharding key: user_id (90% queries carry it). Algorithm: db = user_id % 4, table = (user_id / 4) % 16 (4 DBs, 16 tables each). Gene injection: orderId embeds user_id % 64 for direct routing.
Four Hard Problems
Distributed ID: Snowflake + local segment cache (batch 1000), clock rollback handling (UidGenerator).
Distributed TX: Prefer eventual consistency (local message table + MQ); Seata TCC for strong needs.
Cross-shard: Ban cross-shard JOIN; duplicate fields or wide table; ES for global search; gene method for orderId lookup.
Online Scaling: Double expansion (power-of-2 logical shards). Pre-create 64 logical shards mapped to 4 physical DBs. To add 4 DBs, remap logical shards, migrate via Canal CDC, dual-write, verify, cutover. Zero code change.
Middleware
ShardingSphere-JDBC (client-side, lightweight, Java) preferred over MyCat (proxy, single-point risk).
API Anti-Brush / Rate Limiting
Layered Defense
Nginx: limit_req or lua-resty-limit-traffic by IP+path.
Gateway (Spring Cloud Gateway): Redis token bucket per user/IP via RequestRateLimiter filter.
Application: fine-grained per core endpoint (e.g., order 10 req/s).
Algorithm Choice
Token bucket (allows burst) > sliding window log (precise but heavy) > sliding window counter (balanced) > fixed window (boundary burst) > leaky bucket (strict smoothing). Implement distributed token bucket with Redis+Lua for atomicity.
-- Token bucket Lua
local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local rate = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local bucket = redis.call('HMGET', key, 'tokens', 'last_time')
local tokens = tonumber(bucket[1]) or capacity
local last_time = tonumber(bucket[2]) or now
local delta = math.max(0, now - last_time) * rate / 1000
tokens = math.min(capacity, tokens + delta)
if tokens >= 1 then
tokens = tokens - 1
redis.call('HMSET', key, 'tokens', tokens, 'last_time', now)
redis.call('EXPIRE', key, 60)
return 1
else return 0 endAdvanced Anti-Brush
Multi-dimension: IP + userId + device fingerprint (Canvas/WebGL).
Action sequence anomalies (login→coupon→order in short time) → block.
Soft limit: CAPTCHA; hard limit: 429 with friendly msg; malicious: dynamic Nginx blacklist via ngx.shared.DICT.
Performance: Lua atomic ops, async local counter flush (100ms batch).
Monitoring: Prometheus rate_limited_total{uri,userId}, Grafana dashboards, alerts.
Config Center Design
Core: Manage, Push, Stable
4-layer: Config UI → Config Service (MySQL+Redis) → Push Service (long-polling) → Client SDK (local snapshot + memory).
Storage
Tables: app, config (key, value, env, version), config_history. Version column enables incremental push: SELECT ... WHERE version > ?.
Push Mechanism: Long Polling
Client loops /config/v3/poll with local max version. Server uses DeferredResult (30s timeout). On change, return diff immediately; on timeout, return empty, client re-polls. Compatible with firewalls, near real-time.
Client SDK Resilience
Start: read local snapshot snapshot/{app}_{env}.cache → instant start.
Background: long-poll updates, overwrite memory + snapshot.
Disaster: if config center down, keep using snapshot, retry indefinitely.
Rollback: keep currentConfig and lastStableConfig; one-click revert via JMX.
Advanced
Gray release: tag config with gray_percent=10% or IP list; server evaluates client IP.
Approval flow: high-risk configs (DB password) require ticket → lead approve → callback publish.
Audit: config_history logs who/when/old/new.
RBAC: dev sees dev, ops sees prod, admin manages approval.
Stack
Spring Boot 3, MySQL 8 + Redis, Spring MVC DeferredResult, custom lightweight SDK, Micrometer+Prometheus. Or wrap Nacos/Apollo with thin adapter.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
