The 8 Characters in a Text Message Reveal Hidden Challenges in Short‑Link System Design

Why short links are needed

Platform character limits (e.g., early Twitter 140‑char limit) and SMS billing rules (messages over 70 Chinese characters are split and billed twice) make long URLs costly. Long URLs with many query parameters are often blocked by SMS gateway filters, while short links avoid this.

Two‑step request flow

Browser → GET /aB3xZ7 (short link) → 302 Location: https://xxx.com/real?... → Browser follows → Target page

Using HTTP 302 (temporary redirect) forces every request through the short‑link server, enabling click statistics, A/B routing and dynamic target changes. A 301 (permanent) would cache the redirect in the browser and prevent later statistics or URL updates.

Core problem: short‑link key generation

Two approaches are common:

1. Hash‑based (MurmurHash)

String longUrl = "https://www.example.com/very-long-path?params=xxx";
long hashCode = MurmurHash3.hash64(longUrl);
String shortKey = toBase62(Math.abs(hashCode)).substring(0, 8);

MurmurHash is preferred over MD5/SHA for speed and low collision rate. The hash is converted to base‑62 (0‑9, a‑z, A‑Z), yielding 6‑8 characters (62⁶ ≈ 5.68 × 10⁹, 62⁸ > 2 × 10¹⁴). Collisions are handled by re‑hashing with a special suffix. For massive scales a Bloom filter (≈125 MB for 1 billion keys) pre‑filters existence to avoid DB hits.

2. ID + Base62 (industry mainstream)

Maintain a global incremental ID and convert the decimal ID to base‑62.

private static final String CHARS = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
public static String toBase62(long id) {
    if (id == 0) return "0";
    StringBuilder sb = new StringBuilder();
    while (id > 0) {
        int remainder = (int)(id % 62);
        sb.append(CHARS.charAt(remainder));
        id /= 62;
    }
    return sb.reverse().toString();
}

Example: ID = 1234567890 → base62 "1lSrPk" (after reversal). Fixed‑length keys are achieved by starting the ID at a large offset (e.g., 10 000 000 000) or by left‑padding to 8 characters and applying a Feistel‑style permutation to hide the monotonic pattern.

Architecture evolution of the ID generator

Generation 1 – MySQL AUTO_INCREMENT: Simple but DB write becomes a bottleneck (~5 k–8 k QPS).

Generation 2 – Segment (batch) mode: Services such as Meituan Leaf or Didi TinyID pre‑fetch a block of IDs (e.g., step = 1 000) into memory, reducing DB writes to one per block. Dual‑buffering smooths the TP999 latency spike.

Generation 3 – Snowflake (clock‑based): 64‑bit ID composed of sign (1 bit) + timestamp (41 bit) + machine ID (10 bit) + sequence (12 bit), supporting ~4.1 M IDs/s. Requires careful handling of clock rollback; Meituan Leaf adds ZooKeeper‑based machine‑ID allocation and multi‑layer fallback for rollbacks.

Overall system architecture

Write path: long URL → validation (domain blacklist, protocol check, private‑IP filter) → Bloom filter check → ID generator → base62 (optional permutation) → store in MySQL + Redis → return short link.

Read path: short key → rate‑limit → Redis (three‑level cache: Nginx local → Redis → MySQL) → 302 redirect → async click logging.

Why caching matters – three real‑world failure scenarios

Double‑11 promotion generates 5 million clicks in 10 minutes; DB QPS ceiling (~30 k) is exceeded.

Hot links receive millions of hits daily while long‑tail links are rarely accessed; using a single DB wastes resources.

Cache expiration spikes cause cache breakdown (mass DB load) and cache penetration (massive requests for non‑existent keys).

Three‑tier cache design

Level 1 – Nginx (or OpenResty) local cache: proxy_cache stores hot mappings, bypassing the Java layer entirely.

Level 2 – Redis: KV of short_key → long_url, QPS > 100 k on a single node.

Level 3 – MySQL read‑replica: Fallback when both caches miss; unique index on short_key guarantees fast lookup.

Mitigations:

Cache penetration – Bloom filter pre‑check; cache empty value with short TTL.

Cache breakdown – Do not set TTL for hot keys; use a distributed lock (e.g., Redisson) for cache rebuild.

Cache avalanche – Randomized TTL (base + random(0,600 s)).

Security & reliability

Malicious URL injection – filter blacklisted domains, enforce HTTP/HTTPS, block private IPs, optionally call Google Safe Browsing or VirusTotal.

Abuse of the generation API – multi‑dimensional rate limiting (global QPS, per‑user QPM, per‑IP QPM) plus CAPTCHA for unauthenticated users.

Key enumeration – apply Feistel‑style ID obfuscation or generate high‑entropy random keys (≈16 chars, 62¹⁶ ≈ 4.7 × 10²⁸) for private links.

Click‑fraud – deduplicate clicks per IP+UA within 5 min, process logs asynchronously, and use device fingerprinting for high‑value actions.

Engineering pitfalls

Duplicate short links for the same long URL – check cache/DB first; use INSERT IGNORE or a unique index to avoid redundancy.

Cache breakdown on hot links – keep hot keys permanent or protect rebuild with a lock.

Segment exhaustion during DB outage – set step size to 600 × peak QPS so the in‑memory buffer lasts 10‑20 minutes.

Snowflake machine‑ID allocation in containers – use ZooKeeper sequential nodes and persist the worker ID locally.

Missing internal‑IP filter leads to SSRF attacks – reject private IP ranges during URL validation.

Capacity estimation

Assume 100 million new links per day, peak write QPS = 50 k, read/write ratio = 100:1.

Record size ≈ 280 B (short_key 16 B + long_url 200 B + other 64 B)
1 day: 100 M × 280 B ≈ 28 GB
3 years: ≈ 30 TB → sharding or NoSQL required
Redis for hot 20 % (20 M keys) ≈ 4 GB → 128 GB cluster is ample
Segment step = 5 k × 600 s = 30 M IDs → DB writes ≈ 0.0017 writes/s

Sharding strategy

When short_link_map exceeds ~50 M rows, split by hashing short_key and taking modulo of a power‑of‑two shard count (e.g., 32, 64) to enable easy expansion.

Final comparison: hash vs ID schemes

Implementation complexity: Hash – low; ID – medium (maintain ID service).

Collision handling: Hash – re‑hash + Bloom filter; ID – none (unique IDs).

Idempotency for same long URL: Hash – same key (deterministic); ID – different keys (requires de‑duplication).

Predictability: Hash – unordered, more secure; ID – monotonic, requires obfuscation.

Enumeration difficulty: Hash – high; ID – low unless mixed with obfuscation.

Suitable scale: Hash – small‑to‑medium; ID – large‑scale, high‑concurrency.

One‑sentence advice: use the hash scheme for small‑to‑medium traffic where idempotency matters; adopt segment‑mode ID generation for large‑scale, high‑concurrency scenarios, and always add Feistel‑style mixing or random high‑entropy keys when security against enumeration is required.