How to Build a Million-User Ticket Spike System with Go, Nginx, and Redis

12306 Ticket Spike: Lessons from Extreme Concurrency

During holidays, millions of users compete for train tickets, creating a massive QPS that challenges any flash‑sale system. By analyzing the 12306 backend architecture, we illustrate how to design a service that can handle 1 million concurrent users buying 10 000 tickets while remaining stable.

1. Large‑Scale High‑Concurrency Architecture

High‑concurrency systems typically use distributed clusters with multiple load‑balancing layers and disaster‑recovery mechanisms (dual data centers, node fault tolerance, server backup) to ensure high availability. Traffic is balanced across servers according to capacity and configuration.

1.1 Load‑Balancing Overview

The request flow passes through three layers of load balancers:

OSPF (Open Shortest Path First) is an interior gateway protocol that builds a link‑state database and computes shortest‑path trees; it can perform load balancing across equal‑cost paths.

LVS (Linux Virtual Server) provides IP‑level load balancing and hides server failures, forming a high‑performance virtual server.

Nginx, a high‑performance HTTP reverse proxy, supports round‑robin, weighted round‑robin, and IP‑hash load balancing. The example below focuses on weighted round‑robin.

1.2 Nginx Weighted Round‑Robin Demo

The upstream module implements weighted round‑robin. The following configuration assigns weights 1‑4 to four backend services listening on ports 3001‑3004.

upstream load_rule {
    server 127.0.0.1:3001 weight=1;
    server 127.0.0.1:3002 weight=2;
    server 127.0.0.1:3003 weight=3;
    server 127.0.0.1:3004 weight=4;
}
server {
    listen 80;
    server_name load_balance.com www.load_balance.com;
    location / {
        proxy_pass http://load_rule;
    }
}

A simple Go program runs four HTTP services on ports 3001‑3004 to handle ticket‑purchase requests.

package main

import (
    "net/http"
    "os"
    "strings"
)

func main() {
    http.HandleFunc("/buy/ticket", handleReq)
    http.ListenAndServe(":3001", nil)
}

func handleReq(w http.ResponseWriter, r *http.Request) {
    failedMsg := "handle in port:"
    writeLog(failedMsg, "./stat.log")
}

func writeLog(msg string, logPath string) {
    fd, _ := os.OpenFile(logPath, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0644)
    defer fd.Close()
    content := strings.Join([]string{msg, "
"}, "3001")
    buf := []byte(content)
    fd.Write(buf)
}

Using ab for load testing shows that each port receives requests proportional to its weight, confirming the effectiveness of weighted load balancing.

2. Spike System Design Choices

To keep the ticket‑spike service stable under extreme load, we must prevent overselling and underselling while handling the three core stages: order creation, inventory deduction, and payment.

2.1 Order‑Then‑Deduct Inventory

Creating an order first and then deducting inventory guarantees no oversell but incurs heavy DB I/O and risks undersell if users abandon payment.

2.2 Pay‑Then‑Deduct Inventory

Deducting after payment avoids undersell but can cause oversell because many orders may be created before inventory runs out, and it still involves heavy DB I/O.

2.3 Pre‑Deduct Inventory (Best Choice)

Pre‑deducting inventory ensures no oversell; order creation is asynchronous via a message queue (e.g., Kafka). If a user does not pay within a timeout (e.g., five minutes), the order expires and the inventory is returned.

3. The Art of Inventory Deduction

Pre‑deduction is implemented by allocating a portion of inventory to each machine’s memory, reducing the need for frequent DB writes. A cluster of 100 machines each holds 100 tickets, totaling 10 000 tickets.

To handle node failures, each machine also reserves a “buffer” inventory. A centralized Redis store maintains the global inventory; local deductions succeed first, then a remote Redis deduction confirms the transaction. If a node crashes, its buffer inventory can be reclaimed by other nodes.

Redis, with its single‑threaded nature, can handle up to 100 k QPS. The buffer size must be tuned to balance fault tolerance against Redis load.

4. Code Demonstration (Go)

4.1 Initialization

The init function sets local inventory, Redis connection pool, and a channel used as a lightweight distributed lock.

package localSpike

type LocalSpike struct {
    LocalInStock     int64
    LocalSalesVolume int64
}

package remoteSpike

type RemoteSpikeKeys struct {
    SpikeOrderHashKey string // Redis hash key for orders
    TotalInventoryKey string // Total tickets key
    QuantityOfOrderKey string // Sold tickets key
}

func NewPool() *redis.Pool {
    return &redis.Pool{MaxIdle: 10000, MaxActive: 12000, Dial: func() (redis.Conn, error) {
        return redis.Dial("tcp", ":6379")
    }}
}

func init() {
    localSpike = localSpike2.LocalSpike{LocalInStock: 150, LocalSalesVolume: 0}
    remoteSpike = remoteSpike2.RemoteSpikeKeys{SpikeOrderHashKey: "ticket_hash_key", TotalInventoryKey: "ticket_total_nums", QuantityOfOrderKey: "ticket_sold_nums"}
    redisPool = remoteSpike2.NewPool()
    done = make(chan int, 1)
    done <- 1
}

4.2 Local and Remote Deduction

func (spike *LocalSpike) LocalDeductionStock() bool {
    spike.LocalSalesVolume++
    return spike.LocalSalesVolume < spike.LocalInStock
}

const LuaScript = `
    local ticket_key = KEYS[1]
    local ticket_total_key = ARGV[1]
    local ticket_sold_key = ARGV[2]
    local ticket_total_nums = tonumber(redis.call('HGET', ticket_key, ticket_total_key))
    local ticket_sold_nums = tonumber(redis.call('HGET', ticket_key, ticket_sold_key))
    if ticket_total_nums >= ticket_sold_nums then
        return redis.call('HINCRBY', ticket_key, ticket_sold_key, 1)
    end
    return 0
`

func (r *RemoteSpikeKeys) RemoteDeductionStock(conn redis.Conn) bool {
    lua := redis.NewScript(1, LuaScript)
    result, err := redis.Int(lua.Do(conn, r.SpikeOrderHashKey, r.TotalInventoryKey, r.QuantityOfOrderKey))
    if err != nil { return false }
    return result != 0
}

4.3 HTTP Service

package main

func main() {
    http.HandleFunc("/buy/ticket", handleReq)
    http.ListenAndServe(":3005", nil)
}

func handleReq(w http.ResponseWriter, r *http.Request) {
    redisConn := redisPool.Get()
    <-done
    var logMsg string
    if localSpike.LocalDeductionStock() && remoteSpike.RemoteDeductionStock(redisConn) {
        util.RespJson(w, 1, "抢票成功", nil)
        logMsg = "result:1,localSales:" + strconv.FormatInt(localSpike.LocalSalesVolume, 10)
    } else {
        util.RespJson(w, -1, "已售罄", nil)
        logMsg = "result:0,localSales:" + strconv.FormatInt(localSpike.LocalSalesVolume, 10)
    }
    done <- 1
    writeLog(logMsg, "./stat.log")
}

4.4 Performance Test

Using ApacheBench ( ab -n 10000 -c 100 http://127.0.0.1:3005/buy/ticket) the single‑machine service handled over 4 000 requests per second, with latency around 23 ms, demonstrating that the design scales well when replicated across many nodes.

Requests per second:    4275.96 [#/sec] (mean)
Time per request:       23.387 [ms] (mean)
...

5. Summary

The spike system combines load‑balancing, local memory inventory, Redis atomic deduction, and Go’s native concurrency to avoid database bottlenecks, prevent oversell/undersell, and tolerate node failures. Key takeaways are effective load distribution, leveraging asynchronous processing, and maximizing CPU utilization with goroutines.