How to Build a Million‑User Ticket‑Spiking System with Nginx, Go, and Redis

Background

During holidays, millions of users in China rush to buy train tickets on the 12306 platform, creating an extreme spike in traffic that challenges any second‑kill system.

High‑Concurrency System Architecture

Large‑scale systems typically use distributed clusters with multiple layers of load balancing and disaster‑recovery mechanisms (dual data centers, node fault tolerance, server backup) to ensure high availability.

Load‑Balancing Overview

The article introduces three common load‑balancing methods:

OSPF (Open Shortest Path First) – an interior gateway protocol that builds a link‑state database and computes shortest paths, allowing manual cost adjustments for traffic engineering.

LVS (Linux Virtual Server) – an IP‑level load balancer that distributes requests across a pool of real servers and masks server failures.

Nginx – a high‑performance HTTP reverse proxy that supports round‑robin, weighted round‑robin, and IP‑hash scheduling.

Nginx Weighted Round‑Robin Example

# Configure load balancing
upstream load_rule {
    server 127.0.0.1:3001 weight=1;
    server 127.0.0.1:3002 weight=2;
    server 127.0.0.1:3003 weight=3;
    server 127.0.0.1:3004 weight=4;
}
server {
    listen 80;
    server_name load_balance.com www.load_balance.com;
    location / {
        proxy_pass http://load_rule;
    }
}

Order Processing Flow

The typical ticket‑sale process involves three stages: order creation, inventory deduction, and user payment. The article discusses three sequencing strategies and their drawbacks under extreme concurrency.

Strategy 1 – Order → Deduct Inventory

Creating an order first and then deducting inventory guarantees no overselling but incurs heavy database I/O and is vulnerable to malicious users who place orders without paying.

Strategy 2 – Payment → Deduct Inventory

Deducting inventory after payment avoids overselling but can cause “oversell” when many orders are created without payment, and still suffers from high DB load.

Strategy 3 – Pre‑Deduction (Reserve Stock)

Reserve stock in memory, deduct locally, and asynchronously create orders. This reduces DB I/O, improves response time, and, combined with a timeout‑based order expiration, prevents both overselling and underselling.

Implementation Details

Local Stock Deduction (Go)

package localSpike
// LocalDeductionStock returns true if sales volume is still below local stock
func (spike *LocalSpike) LocalDeductionStock() bool {
    spike.LocalSalesVolume = spike.LocalSalesVolume + 1
    return spike.LocalSalesVolume < spike.LocalInStock
}

Remote Stock Deduction (Redis + Lua)

package remoteSpike
const LuaScript = `
local ticket_key = KEYS[1]
local ticket_total_key = ARGV[1]
local ticket_sold_key = ARGV[2]
local ticket_total_nums = tonumber(redis.call('HGET', ticket_key, ticket_total_key))
local ticket_sold_nums = tonumber(redis.call('HGET', ticket_key, ticket_sold_key))
if (ticket_total_nums >= ticket_sold_nums) then
    return redis.call('HINCRBY', ticket_key, ticket_sold_key, 1)
end
return 0
`
func (r *RemoteSpikeKeys) RemoteDeductionStock(conn redis.Conn) bool {
    lua := redis.NewScript(1, LuaScript)
    result, err := redis.Int(lua.Do(conn, r.SpikeOrderHashKey, r.TotalInventoryKey, r.QuantityOfOrderKey))
    if err != nil {
        return false
    }
    return result != 0
}

Service Initialization

Initialize local stock, Redis connection pool, and a channel of size 1 to act as a distributed lock.

HTTP Handler

func handleReq(w http.ResponseWriter, r *http.Request) {
    redisConn := redisPool.Get()
    <-done // acquire lock
    if localSpike.LocalDeductionStock() && remoteSpike.RemoteDeductionStock(redisConn) {
        util.RespJson(w, 1, "抢票成功", nil)
        LogMsg = "result:1,localSales:" + strconv.FormatInt(localSpike.LocalSalesVolume, 10)
    } else {
        util.RespJson(w, -1, "已售罄", nil)
        LogMsg = "result:0,localSales:" + strconv.FormatInt(localSpike.LocalSalesVolume, 10)
    }
    done <- 1 // release lock
    writeLog(LogMsg, "./stat.log")
}

Performance Testing

Using ApacheBench (ab) with 10,000 requests and 100 concurrent connections, the single‑machine service achieved over 4,000 requests per second, demonstrating that the combination of Nginx load balancing, local in‑memory stock, and Redis atomic operations can handle massive traffic.

Key Takeaways

Distribute traffic with weighted load balancing to let each server handle a manageable slice of the load.

Leverage in‑memory stock deduction and asynchronous order creation to avoid costly database transactions.

Use Redis Lua scripts for atomic remote stock updates, ensuring no overselling while tolerating server failures via buffer stock.

Go’s lightweight goroutines and channel‑based locking provide efficient concurrency control.