How Go 1.25’s Trace Flight Recorder Enables Low‑Overhead Production Debugging

Tracing vs Flight Recording

Traditional tracing records every event for the entire program lifetime, producing large trace files and incurring high overhead. Flight Recording keeps only the most recent events in a circular buffer, discarding older data automatically. This bounded buffer makes it suitable for continuous production use.

Configuration

The core struct is trace.FlightRecorderConfig with two fields: MinAge – minimum duration an event must stay in the buffer before it can be dropped (e.g., 5 * time.Second keeps data for at least five seconds). MaxBytes – maximum size of the circular buffer (e.g., 3 * 1024 * 1024 limits the buffer to 3 MB).

These are advisory limits; the runtime may keep slightly more data.

Step‑by‑step demonstration

1. Simple HTTP server with a heavy endpoint

package main

import (
    "log"
    "net/http"
    "math/big"
    "crypto/sha256"
    "strconv"
    "runtime"
)

func pow(targetBits int) [32]byte {
    target := big.NewInt(1)
    target.Lsh(target, uint(256-targetBits))
    var hashInt big.Int
    var hash [32]byte
    nonce := 0
    for {
        data := "hello world " + strconv.Itoa(nonce)
        hash = sha256.Sum256([]byte(data))
        hashInt.SetBytes(hash[:])
        if hashInt.Cmp(target) == -1 {
            break
        }
        nonce++
        if nonce%100 == 0 {
            runtime.Gosched()
        }
    }
    return hash
}

func handler(w http.ResponseWriter, r *http.Request) {
    if r.URL.Path == "/heavy" {
        _ = pow(20) // simulate CPU load
    }
    w.Write([]byte("Hello, world!"))
}

func main() {
    http.HandleFunc("/", handler)
    log.Fatal(http.ListenAndServe(":8080", nil))
}

2. Start a Flight Recorder

package main

import (
    "log"
    "net/http"
    "time"
    "runtime/trace"
)

var recorder *trace.FlightRecorder

func main() {
    cfg := trace.FlightRecorderConfig{MinAge: 5 * time.Second, MaxBytes: 3 * 1024 * 1024}
    recorder = trace.StartFlightRecorder(cfg)
    defer recorder.Stop()

    http.HandleFunc("/", handler)
    log.Fatal(http.ListenAndServe(":8080", nil))
}

3. Add a performance trigger that writes the latest buffer to a file

package main

import (
    "log"
    "net/http"
    "time"
    "runtime/trace"
    "os"
    "strings"
    "math/rand"
)

func handlerWithTrigger(w http.ResponseWriter, r *http.Request) {
    // Random difficulty between 10 and 30 bits
    hash := pow(rand.Intn(20) + 10)
    // Trigger when the first six hex characters are zero
    if strings.HasPrefix(fmt.Sprintf("%x", hash[:]), "000000") {
        f, err := os.Create("trace.out")
        if err != nil {
            log.Println("failed to create trace file:", err)
            return
        }
        defer f.Close()
        if _, err := recorder.WriteTo(f); err != nil {
            log.Println("failed to write trace data:", err)
        }
    }
    w.Write([]byte(fmt.Sprintf("%x", hash[:])))
}

func main() {
    cfg := trace.FlightRecorderConfig{MinAge: 5 * time.Second, MaxBytes: 3 * 1024 * 1024}
    recorder = trace.NewFlightRecorder(cfg)
    if err := recorder.Start(); err != nil {
        log.Fatalf("failed to start FlightRecorder: %v", err)
    }
    defer recorder.Stop()

    http.HandleFunc("/", handlerWithTrigger)
    log.Fatal(http.ListenAndServe(":8080", nil))
}

After a trigger fires, the file trace.out can be inspected with: go tool trace trace.out The tool opens a browser visualising goroutine lifecycles, CPU utilisation per logical processor, and heap allocation/GC patterns.

Analyzing trace data

Goroutine activity – creation, execution, and blocking.

CPU usage – per‑processor utilisation over time.

Heap usage – allocation rates and garbage‑collection behaviour.

Core API

All functionality lives in the runtime/trace package.

type FlightRecorder struct { /* internal fields */ }

func NewFlightRecorder() *FlightRecorder
func (fr *FlightRecorder) SetMinAge(d time.Duration)
func (fr *FlightRecorder) SetMaxBytes(b uint64)
func (fr *FlightRecorder) Start() error
func (fr *FlightRecorder) Stop() error
func (fr *FlightRecorder) Enabled() bool
func (fr *FlightRecorder) WriteTo(w io.Writer) (int64, error)

The recorder hooks into the runtime’s tracing system (e.g., the ProcSteal event) to capture scheduler‑level events such as P‑stealing, including the stolen M’s ID, the event sequence number, and the previous M’s ID.

Design background

The feature originated from Go issue #63185 (https://github.com/golang/go/issues/63185). It mirrors Java Flight Recorder’s circular‑buffer approach, allowing a program to retain only the most recent seconds of execution. The implementation became stable in Go 1.22 (issue #60773: https://github.com/golang/go/issues/60773) after Go 1.21 reduced tracing overhead, making continuous recording feasible in production clusters.

Design documentation resides at https://go.googlesource.com/proposal/+/ac09a140c3d26f8bb62cbad8969c8b154f93ead6/design/60773-execution-tracer-overhaul.md, which describes the partitioned trace format introduced in Go 1.22 and extended in Go 1.25.

Typical use cases

Production debugging – capture context around rare, hard‑to‑reproduce errors without the cost of a full trace.

Performance monitoring – diagnose intermittent latency spikes that only appear under specific conditions.

Memory‑constrained environments – low overhead and bounded buffer size suit devices with limited resources.