How to Build a Million-User Ticket‑Sniping System with Go, Nginx, and Redis

Ticket Sniping on 12306: Lessons from Extreme Concurrency

During peak periods such as Chinese New Year, hundreds of millions of users compete for train tickets, creating a massive spike in QPS that rivals any flash‑sale system.

Large‑Scale High‑Concurrency Architecture

Typical high‑concurrency systems use distributed clusters with multiple layers of load balancing and disaster‑recovery mechanisms (dual data centers, node fault tolerance, backup servers) to ensure high availability.

Below is a simplified diagram of such an architecture:

Load‑Balancing Overview

The user request passes through three layers of load balancers:

OSPF (Open Shortest Path First) – an interior gateway protocol that builds a link‑state database and computes shortest‑path trees; costs are inversely proportional to bandwidth and can be manually overridden.

LVS (Linux Virtual Server) – a cluster technology that distributes requests across servers and masks server failures, providing a high‑throughput virtual server.

Nginx – a high‑performance HTTP reverse proxy that offers several load‑balancing methods.

We focus on Nginx weighted round‑robin configuration and testing.

Nginx Weighted Round‑Robin Demo

The upstream module defines server weights (1‑4) for ports 3001‑3004:

# Configuration for load balancing
upstream load_rule {
    server 127.0.0.1:3001 weight=1;
    server 127.0.0.1:3002 weight=2;
    server 127.0.0.1:3003 weight=3;
    server 127.0.0.1:3004 weight=4;
}
...
server {
    listen 80;
    server_name load_balance.com www.load_balance.com;
    location / {
        proxy_pass http://load_rule;
    }
}

Requests are logged to ./stat.log and tested with ApacheBench (AB).

ab -n 1000 -c 100 http://www.load_balance.com/buy/ticket

The log shows request distribution 100, 200, 300, 400 across the four ports, matching the configured weights.

Flash‑Sale System Design Choices

Three typical order‑processing stages are considered:

Order‑Create → Decrease‑Stock

Creating an order first and then decreasing stock guarantees no overselling but incurs heavy DB I/O and suffers from malicious orders that never pay.

Payment → Decrease‑Stock

Decreasing stock after payment avoids lost sales but can cause overselling under extreme concurrency.

Pre‑Deduction (Reserve Stock)

Reserve stock locally, then asynchronously create orders. If payment does not occur within a timeout, the reserved stock is released back to the pool. This reduces DB I/O and improves response time.

Optimizing Stock Deduction

Local in‑memory stock deduction avoids frequent database writes. However, a single machine cannot handle millions of requests, so we combine local deduction with a centralized Redis stock counter.

Architecture diagram (simplified):

Redis stores total inventory and sold count in a hash. After a successful local deduction, the service atomically decrements the Redis counter using a Lua script.

const LuaScript = `
    local ticket_key = KEYS[1]
    local ticket_total_key = ARGV[1]
    local ticket_sold_key = ARGV[2]
    local ticket_total_nums = tonumber(redis.call('HGET', ticket_key, ticket_total_key))
    local ticket_sold_nums = tonumber(redis.call('HGET', ticket_key, ticket_sold_key))
    if (ticket_total_nums >= ticket_sold_nums) then
        return redis.call('HINCRBY', ticket_key, ticket_sold_key, 1)
    end
    return 0
`
func (RemoteSpikeKeys *RemoteSpikeKeys) RemoteDeductionStock(conn redis.Conn) bool {
    lua := redis.NewScript(1, LuaScript)
    result, err := redis.Int(lua.Do(conn, RemoteSpikeKeys.SpikeOrderHashKey, RemoteSpikeKeys.TotalInventoryKey, RemoteSpikeKeys.QuantityOfOrderKey))
    if err != nil { return false }
    return result != 0
}

Initial Redis state:

hmset ticket_hash_key "ticket_total_nums" 10000 "ticket_sold_nums" 0

Go Implementation

Go’s native concurrency model is used to run four HTTP services (ports 3001‑3004). The init function prepares local stock, Redis keys, and a channel that acts as a distributed lock.

package localSpike
func (spike *LocalSpike) LocalDeductionStock() bool {
    spike.LocalSalesVolume = spike.LocalSalesVolume + 1
    return spike.LocalSalesVolume < spike.LocalInStock
}

The main handler performs:

Acquire the channel lock.

Execute LocalDeductionStock() and RemoteDeductionStock().

Return JSON indicating success or “sold out”.

Log the result.

func handleReq(w http.ResponseWriter, r *http.Request) {
    redisConn := redisPool.Get()
    <-done
    if localSpike.LocalDeductionStock() && remoteSpike.RemoteDeductionStock(redisConn) {
        util.RespJson(w, 1, "抢票成功", nil)
        LogMsg = "result:1,localSales:" + strconv.FormatInt(localSpike.LocalSalesVolume, 10)
    } else {
        util.RespJson(w, -1, "已售罄", nil)
        LogMsg = "result:0,localSales:" + strconv.FormatInt(localSpike.LocalSalesVolume, 10)
    }
    done <- 1
    writeLog(LogMsg, "./stat.log")
}

Benchmarking with AB (10 000 requests, 100 concurrent) on a low‑end Mac yields ~4 300 requests per second, confirming that a single node can handle several thousand QPS. Logs show a smooth distribution of requests across ports.

Key Takeaways

1. Load balancing and sharding distribute traffic across many machines, allowing the system to scale horizontally.

2. Combining local in‑memory stock with a centralized Redis counter eliminates most DB I/O while preserving atomicity via Lua scripts.

3. Go’s goroutine‑based concurrency and channel‑based locking provide a simple yet effective way to serialize critical sections without sacrificing throughput.

These techniques together enable a ticket‑sniping service to sustain extreme traffic without overselling or significant loss of sales.