How to Achieve Full Observability for Go Apps Without Intrusive Agents

Observability Overview

Observability is built on four data pillars—metrics, logs, tracing, and continuous profiling—and enables capabilities such as monitoring, problem analysis, and system diagnosis by correlating these data sources.

SDK Approach

Java can use bytecode enhancement for non‑intrusive monitoring, but Go compiles to a static binary, making such dynamic instrumentation impossible. The SDK solution relies on the OpenTelemetry Go SDK, requiring developers to add manual instrumentation points.

package main

import (
    "context"
    "fmt"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/sdk/trace"
    "io"
    "net/http"
)

func init() {
    tp := trace.NewTracerProvider()
    otel.SetTracerProvider(tp)
}

func main() {
    for {
        tracer := otel.GetTracerProvider().Tracer("")
        ctx, span := tracer.Start(context.Background(), "Client/User defined span")
        req, err := http.NewRequestWithContext(ctx, "GET", "http://otel-server:9000/http-service1", nil)
        if err != nil { fmt.Println(err.Error()); continue }
        client := &http.Client{}
        resp, err := client.Do(req)
        if err != nil { fmt.Println(err.Error()); continue }
        defer resp.Body.Close()
        b, err := io.ReadAll(resp.Body)
        if err != nil { fmt.Println(err.Error()); continue }
        fmt.Println(string(b))
        span.SetAttributes(attribute.String("client", "client-with-ot"))
        span.SetAttributes(attribute.Bool("user.defined", true))
        span.End()
    }
}

The code creates a TraceProvider, starts a span for each request, records attributes, and ends the span after the request completes. Complex applications need instrumentation at every external call and proper context propagation.

func testContext() {
    tracer := otel.Tracer("app-tracer")
    opts := append([]trace.SpanStartOption{}, trace.WithSpanKind(trace.SpanKindServer))
    rootCtx, rootSpan := tracer.Start(context.Background(), getRandomSpanName(), opts...)
    if !rootSpan.SpanContext().IsValid() { panic("invalid root span") }
    go func() {
        opts1 := append([]trace.SpanStartOption{}, trace.WithSpanKind(trace.SpanKindInternal))
        _, subSpan1 := tracer.Start(rootCtx, getRandomSpanName(), opts1...)
        defer subSpan1.End()
    }()
    go func() {
        opts2 := append([]trace.SpanStartOption{}, trace.WithSpanKind(trace.SpanKindInternal))
        _, subSpan2 := tracer.Start(rootCtx, getRandomSpanName(), opts2...)
        defer subSpan2.End()
    }()
    rootSpan.End()
}

Incorrect context propagation breaks the trace chain.

eBPF Approach

eBPF is a high‑performance virtual machine inside the Linux kernel that allows custom programs to be attached to tracepoints, kprobes, or uprobes. Open‑source projects such as Pixie, Beyla, opentelemetry‑go‑instrumentation, and DeepFlow use eBPF for profiling, network monitoring, metric collection, and distributed tracing.

eBPF requires privileged Linux environments and specific kernel versions, and its performance overhead can be higher than in‑process agents because of user‑kernel context switches.

Compile‑time Instrumentation

Alibaba Cloud’s Go Agent (Instgo) injects monitoring code during the build process. By downloading the Instgo tool and prefixing the normal go build command, the binary is automatically instrumented without modifying source code.

# Current build command
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build main.go

# Using Aliyun Go Agent
wget "http://arms-apm-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/instgo/instgo-linux-amd64" -O instgo
chmod +x instgo
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 ./instgo go build main.go

The injected binary provides the same capabilities as Java agents—tracing, metrics, profiling, dynamic configuration, log‑trace correlation, etc.—with about 5 % overhead for 1000 qps and supports more than 40 plugins for common frameworks.

Conclusion

While SDK and eBPF solutions each have strengths, compile‑time instrumentation offers a non‑intrusive, low‑overhead, and production‑ready way to achieve comprehensive observability for Go applications.