Go 1.22’s Powerful Tracing: Low‑Cost, Scalable, and Blackbox Recording Explained

Problem

Historically, Go's runtime/trace package suffered from four major issues: high CPU overhead (10‑20% for many applications), poor scalability that could produce multi‑gigabyte traces, lack of quantitative metrics to decide when to start capturing problematic behavior, and the absence of a public, easy‑to‑use tool for parsing and interpreting trace data.

Low‑Cost Tracing

Work by Felix Geisendörfer and Nick Ripley dramatically reduced the runtime overhead of tracing. In Go 1.22 the cost of generating a trace has dropped to roughly 1‑2% of CPU time, making tracing practical for long‑running services. The improvement comes from optimizing stack‑trace collection during event emission.

Scalable Trace

The trace format is designed for efficient emission, but analyzing large traces still required gigabytes of memory. To keep overhead low while expanding trace coverage, the runtime now supports splitting a trace into independent segments. Each segment behaves like a self‑contained trace, and new data continues seamlessly after a split.

Although the full trace is still loaded into memory by go tool trace, the previous limitation that prevented analysis of very large traces has been removed in Go 1.22.

Blackbox Recorder

An experimental "blackbox" recorder has been added to the golang.org/x/exp/trace package. It continuously records trace data with minimal overhead and can be flushed on demand, enabling post‑mortem analysis of events that occurred before a problem was detected.

fr := trace.NewFlightRecorder()
fr.Start()
var once sync.Once
http.HandleFunc("/my-endpoint", func(w http.ResponseWriter, r *http.Request) {
    start := time.Now()
    doWork(w, r)
    if time.Since(start) > 300*time.Millisecond {
        once.Do(func() {
            var b bytes.Buffer
            _, err = fr.WriteTo(&b)
            if err != nil {
                log.Print(err)
                return
            }
            if err := os.WriteFile("trace.out", b.Bytes(), 0o755); err != nil {
                log.Print(err)
                return
            }
        })
    }
})
log.Fatal(http.ListenAndServe(":8080", nil))

Trace Reader API

An experimental trace reader API is also available in the same package, allowing developers to programmatically inspect trace events. The example below counts how many goroutine state‑transition events represent a wait on the network and reports the percentage of blocked goroutines that are network‑bound.

r, err := trace.NewReader(os.Stdin)
if err != nil {
    log.Fatal(err)
}
var blocked int
var blockedOnNetwork int
for {
    ev, err := r.ReadEvent()
    if err == io.EOF {
        break
    } else if err != nil {
        log.Fatal(err)
    }
    if ev.Kind() == trace.EventStateTransition {
        st := ev.StateTransition()
        if st.Resource.Kind == trace.ResourceGoroutine {
            from, to := st.GoroutineTransition()
            if from.Executing() && to == trace.GoWaiting {
                blocked++
                if strings.Contains(st.Reason, "network") {
                    blockedOnNetwork++
                }
            }
        }
    }
}
p := 100 * float64(blockedOnNetwork) / float64(blocked)
fmt.Printf("%2.3f%% instances of goroutines blocking were to block on the network
", p)

Conclusion

The Go 1.22 release delivers a substantially more efficient, scalable, and extensible tracing experience, with new blackbox recording and a programmable reader API that empower developers to diagnose performance bottlenecks and blocking behavior in production services.