Master Go Debugging: From Heisenbugs to Git Bisect and Ghost Logs

1. The Three Bug Schools in Go

Debugging is said to be twice as hard as writing code; if you use all your cleverness while coding, you must be even smarter when debugging. In Go, bugs are grouped into three "schools":

Copyable (Shaolin) : reproducible bugs that can be fixed with simple logging (e.g., log.Printf) and the Delve debugger.

Heisenbug (Wudang) : bugs that disappear when observed; detect with pprof and some mystical tricks.

Concurrent (Ming) : data‑race bugs caused by unsynchronized goroutine access; detect with go test -race and fix with proper locking.

2. A Full‑Blown (and Ridiculous) Example

Consider an inventory deduction service that looks innocent but contains two bugs:

package main

import (
    "fmt"
    "sync"
)

type Inventory struct {
    count int
    mu    sync.Mutex
}

func (i *Inventory) Deduct(n int) {
    i.count -= n // 🤡 Bug 1: no lock!
    fmt.Printf("Deducted %d, now %d
", n, i.count)
}

func main() {
    inv := &Inventory{count: 100}
    var wg sync.WaitGroup
    for j := 0; j < 10; j++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for i := 0; i < 10; i++ {
                inv.Deduct(1) // 🤡 Bug 2: concurrent call without protection
            }
        }()
    }
    wg.Wait()
    fmt.Printf("Final inventory: %d
", inv.count)
}

Running it shows inconsistent results and a final inventory far from the expected value.

3. Debug Flowchart

发现 bug → 蒙圈三秒 → 是不是我眼睛花了？ → 重跑一次 → 还是错 →
    ├─→ 是不是并发？ → go test -race！
    └─→ 不是并发？ → 是不是 Heisenbug？ → 停下喝口茶，重读代码

Step 1️⃣: Use the Data‑Race Detector

$ go run -race inventory.go
WARNING: DATA RACE
Write at 0x... by goroutine 7:
    main.(*Inventory).Deduct(...)
Found 1 data race (s)
exit status 66

The -race flag works like a paparazzi that catches goroutine misbehaviour. It should not be used in production because of overhead.

Step 2️⃣: Fix with Proper Locking

func (i *Inventory) Deduct(n int) {
    i.mu.Lock()
    defer i.mu.Unlock()
    old := i.count
    i.count -= n
    fmt.Printf("[Goroutine %p] Deduct %d: %d → %d
", goid(), n, old, i.count)
}

Adding a mutex and a defer Unlock eliminates the race, and the program now prints deterministic logs for each goroutine.

Step 3️⃣: Conditional Debug Logging with what

// +build debug
package main
import "github.com/appliedgo/what"

func (i *Inventory) Deduct(n int) {
    i.mu.Lock(); defer i.mu.Unlock()
    what.Happens(i.count, "before deduct")
    old := i.count
    i.count -= n
    what.Happens(i.count, "after deduct")
    fmt.Printf("[Goroutine %p] Deduct %d: %d → %d
", goid(), n, old, i.count)
}

When compiled with go run -tags debug, the what calls emit logs; otherwise they disappear, leaving zero overhead.

Heisenbug Emergency Plan 🚨

Replace fmt.Printf with a lock‑free logger (e.g., zap’s DPanic with async writer).

Record goroutine behavior with go tool trace and inspect the timeline.

Run with GOMAXPROCS=1 to simulate a single‑threaded world; if the bug vanishes, it’s a scheduling issue.

4. Time‑Travel Debugging with git bisect

When a regression appears (e.g., inventory goes negative after a pressure test), use git bisect to locate the offending commit:

$ git bisect start
$ git bisect bad HEAD
$ git bisect good c0ffeeee
# test each checkout
$ go run inventory.go && echo "✅ good" || echo "❌ bad"
# after a few rounds
f00dcafe is the first bad commit

Inspect the diff to see that a lock was removed, confirming the cause.

5. Bonus: GoTutor for Beginners

GoTutor is a web‑based simulator that lets you step forward and backward through program execution, visualizing variables, stacks, and goroutine states without installing Delve.

Conclusion

Prevention > Detection > Fixing is the mantra.

Test‑driven development, clear code, and strategic logging dramatically reduce overtime. log.Printf("📍 here: x=%v", x) remains the most reliable debugging aid.

Happy debugging! 🎉