Why the New OpenTelemetry Golang Agent Beats Manual and eBPF Instrumentation

Background

Go is widely used for cloud‑native applications, but the official OpenTelemetry Go SDK ( opentelemetry-go) requires manual span creation, which becomes cumbersome for large code bases.

Instrumentation Options for Go

Manual SDK instrumentation – developers add spans directly in code.

eBPF‑based automatic instrumentation – captures HTTP, database, RPC calls without code changes but has limitations.

Compile‑time instrumentation – tools such as InstrGen and the OpenTelemetry Golang Agent inject code during compilation.

Manual SDK Example

func parentMethod(ctx context.Context) {
    tracer := otel.Tracer("otel-go-tracer")
    ctx, span := tracer.Start(ctx, "parent span")
    fmt.Println(span.SpanContext().TraceID()) // print TraceID
    span.SetAttributes(attribute.String("key", "value"))
    span.SetStatus(codes.Ok, "Success")
    childMethod(ctx)
    span.End()
}

Manual instrumentation scales linearly with the call graph and is error‑prone when context propagation is missed.

eBPF Automatic Instrumentation

Propagates at most eight HTTP headers.

Requires Linux kernel ≥ 4.4.

Introduces overhead due to frequent user‑kernel transitions (Uprobe).

InstrGen (Compile‑time) Limitations

Slow project updates and sparse documentation.

Supports only a few plugins (no MySQL, Redis, etc.).

Still relies on explicit context.Context passing.

OpenTelemetry Golang Agent (v0.1.0‑RC)

The Agent adds two stages before the normal go build process:

Preprocess : scans third‑party dependencies, matches them with instrumentation rules, and prepares any extra dependencies.

Code Injection : inserts trampoline code at function entry/exit according to the matched rules, then proceeds with the regular build.

Instrumentation rule types:

InstFuncRule – inject code at function entry and exit.

InstStructRule – modify a struct to add a field.

InstFileRule – add a new file to the compilation.

Typical build command: go build -toolexec otelbuild cmd/app The -toolexec flag intercepts the Go toolchain and runs the custom otelbuild instrumenter.

Context Propagation Optimizations

The Agent stores the most recent span in a Goroutine‑Local Storage (GLS) structure. If a span is started with context.Background() or a nil context, the Agent retrieves the latest active span from GLS, preserving trace continuity.

func (tr *tracer) Start(
    ctx context.Context, // query parent span from ctx
    name string,
    options ...trace.SpanStartOption,
) (context.Context, trace.Span) { ... }

GLS maintains a singly‑linked list of spans so that when the current span ends, the next‑newest unclosed span becomes the parent.

Baggage Propagation Optimizations

// Create new Baggage
b := baggage.Baggage{}
m, _ := baggage.NewMember("env", "test")
b, _ = b.SetMember(m)
// Store in context
ctx = baggage.ContextWithBaggage(ctx, b)
// Retrieve later
bag := baggage.FromContext(ctx)

If the incoming context is missing, the Agent falls back to the GLS‑stored baggage, ensuring key‑value propagation even when context.Context is not passed.

Supported Plugins (v0.1.0‑RC)

database/sql – https://pkg.go.dev/database/sql echo – https://github.com/labstack/echo (v4.0.0 – v4.12.0)

gin – https://github.com/gin-gonic/gin (v1.7.0 – v1.10.0)

go-redis – https://github.com/redis/go-redis (v9.0.5 – v9.5.1)

gorm – https://github.com/go-gorm/gorm (v1.22.0 – v1.25.9)

logrus – https://github.com/sirupsen/logrus (v1.5.0 – v1.9.3)

mongodb – https://github.com/mongodb/mongo-go-driver (v1.11.1 – v1.15.2)

mux – https://github.com/gorilla/mux (v1.3.0 – v1.8.1)

net/http – https://pkg.go.dev/net/http zap – https://github.com/uber-go/zap (v1.20.0 – v1.27.0)

Documentation & Resources

How the Agent works: https://github.com/alibaba/opentelemetry-go-auto-instrumentation/blob/main/docs/how-it-works.md

Adding a new rule: https://github.com/alibaba/opentelemetry-go-auto-instrumentation/blob/main/docs/how-to-add-a-new-rule.md

Debugging guide: https://github.com/alibaba/opentelemetry-go-auto-instrumentation/blob/main/docs/how-to-debug.md

Writing plugin tests: https://github.com/alibaba/opentelemetry-go-auto-instrumentation/blob/main/docs/how-to-write-tests-for-plugins.md

Compatibility matrix: https://github.com/alibaba/opentelemetry-go-auto-instrumentation/blob/main/docs/compatibility.md

Performance benchmarks: https://github.com/alibaba/opentelemetry-go-auto-instrumentation/blob/main/example/benchmark/benchmark.md

Demo repository: https://github.com/alibaba/opentelemetry-go-auto-instrumentation/tree/main/example/demo

Roadmap (Future Releases)

Support additional frameworks such as Hertz, Kitex, Elasticsearch.

Add metric collection once the OpenTelemetry metric spec stabilizes.

Expose Go runtime metrics (GC count, memory usage).

Enable continuous CPU/Memory profiling and hotspot detection.

Repository

Source code: https://github.com/alibaba/opentelemetry-go-auto-instrumentation

Illustration