Implementing Graceful (Hot) Restart for Go HTTP Services

While developing an HTTP framework in Go, the author explored how to achieve hot restart – restarting a service without breaking ongoing client requests. A hot restart means that when a running process receives a restart command, it does not terminate immediately; it waits for all current logic to finish before shutting down, then starts a new process that inherits the listening socket.

Hot Restart Principle

The process consists of the following steps:

Listen for a restart signal (e.g., SIGHUP).

When the signal arrives, fork a child process and pass the service's listening socket file descriptor to the child.

The child process receives and starts listening on the inherited socket.

After the child is ready, the parent stops accepting new connections.

The parent exits, completing the restart.

In the Go implementation, SIGHUP is used as the restart signal, while SIGINT and SIGTERM trigger a graceful shutdown.

Go Implementation – Process Startup and Listening

// Start listening
http.HandleFunc("/hello", HelloHandler)
server := &http.Server{Addr: ":8081"}

var err error
if *child {
    fmt.Println("In Child, Listening...")
    f := os.NewFile(3, "")
    listener, err = net.FileListener(f)
} else {
    fmt.Println("In Father, Listening...")
    listener, err = net.Listen("tcp", server.Addr)
}
if err != nil {
    fmt.Printf("Listening failed: %v\n", err)
    return
}

The code shows that the child process opens file descriptor 3 (the socket passed from the parent) and creates a net.Listener from it. The parent creates a normal TCP listener.

Signal Handling

func signalHandler() {
    ch := make(chan os.Signal, 1)
    signal.Notify(ch, syscall.SIGINT, syscall.SIGTERM, syscall.SIGHUP)
    for {
        sig := <-ch
        fmt.Printf("signal: %v\n", sig)
        ctx, _ := context.WithTimeout(context.Background(), 20*time.Second)
        switch sig {
        case syscall.SIGINT, syscall.SIGTERM:
            log.Printf("stop")
            signal.Stop(ch)
            server.Shutdown(ctx)
            fmt.Printf("graceful shutdown\n")
            return
        case syscall.SIGHUP:
            // reload
            log.Printf("restart")
            err := restart()
            if err != nil {
                fmt.Printf("graceful restart failed: %v\n", err)
            }
            updatePidFile()
            server.Shutdown(ctx)
            fmt.Printf("graceful reload\n")
            return
        }
    }
}

The handler listens for termination and restart signals. On termination it performs a graceful shutdown; on SIGHUP it calls the restart function, updates the PID file, and shuts down the current server.

Restart Logic

func restart() error {
    tl, ok := listener.(*net.TCPListener)
    if !ok {
        return fmt.Errorf("listener is not tcp listener")
    }
    f, err := tl.File()
    if err != nil {
        return err
    }
    args := []string{"-child"}
    cmd := exec.Command(os.Args[0], args...)
    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr
    // Pass the socket FD as the first extra file (fd 3 in child)
    cmd.ExtraFiles = []*os.File{f}
    return cmd.Start()
}

The function extracts the underlying file descriptor from the TCP listener, creates a new command with the -child flag, and passes the descriptor via ExtraFiles . In the child process, the descriptor becomes fd 3, which is then turned back into a listener.

Complete Example

package main

import (
    "flag"
    "net"
    "net/http"
    "log"
    "os"
    "os/signal"
    "syscall"
    "golang.org/x/net/context"
    "time"
    "os/exec"
    "fmt"
    "io/ioutil"
    "strconv"
)

var (
    server   *http.Server
    listener net.Listener
    child    = flag.Bool("child", false, "")
)

func init() { updatePidFile() }

func updatePidFile() { /* omitted for brevity */ }
func procExsit(tmpDir string) error { /* omitted for brevity */ }

func main() {
    flag.Parse()
    http.HandleFunc("/hello", HelloHandler)
    server = &http.Server{Addr: ":8081"}
    var err error
    if *child {
        fmt.Println("In Child, Listening...")
        f := os.NewFile(3, "")
        listener, err = net.FileListener(f)
    } else {
        fmt.Println("In Father, Listening...")
        listener, err = net.Listen("tcp", server.Addr)
    }
    if err != nil { fmt.Printf("Listening failed: %v\n", err); return }
    go func() { err = server.Serve(listener); if err != nil { fmt.Printf("server.Serve failed: %v\n", err) } }()
    signalHandler()
    fmt.Printf("signalHandler end\n")
}

func HelloHandler(w http.ResponseWriter, r *http.Request) {
    for i := 0; i < 20; i++ {
        log.Printf("working %v\n", i)
        time.Sleep(1 * time.Second)
    }
    w.Write([]byte("world233333!!!!"))
}

func signalHandler() { /* same as above */ }
func restart() error { /* same as above */ }

This full program demonstrates how to achieve transparent hot restart for a Go HTTP service, handling signals, passing the listening socket to a child process, and using Server.Shutdown for graceful termination (available since Go 1.8).