Mastering End-to-End Tracing in Go Microservices with OpenTracing and Zipkin

OpenTracing Overview

OpenTracing provides a vendor‑agnostic API for distributed tracing. A unique CorrelationID (trace ID) is generated at the start of a request and propagated across service boundaries, allowing the tracing back‑end (e.g., Zipkin or Jaeger) to be swapped without code changes.

Choosing a Tracing Library

Both Zipkin and Jaeger implement the OpenTracing API, so applications can depend on OpenTracing to abstract away the concrete library and avoid lock‑in when a different implementation becomes preferable.

Zipkin Architecture

Recorder – stores raw trace data.

Reporter (collecting agent) – forwards data to the UI.

Tracer – creates spans and trace IDs.

UI – visualises traces.

Client‑side Tracing Example (Go)

The following code creates a global OpenTracing tracer backed by Zipkin, configures an HTTP collector, and builds a gRPC client that automatically injects the trace context into outgoing calls.

const (
    endpoint_url = "http://localhost:9411/api/v1/spans"
    host_url = "localhost:5051"
    service_name_cache_client = "cache service client"
    service_name_call_get = "callGet"
)

func newTracer() (opentracing.Tracer, zipkintracer.Collector, error) {
    collector, err := openzipkin.NewHTTPCollector(endpoint_url)
    if err != nil { return nil, nil, err }
    recorder := openzipkin.NewRecorder(collector, true, host_url, service_name_cache_client)
    tracer, err := openzipkin.NewTracer(recorder, openzipkin.ClientServerSameSpan(true))
    if err != nil { return nil, nil, err }
    opentracing.SetGlobalTracer(tracer)
    return tracer, collector, nil
}

key := "123"
tracer, collector, err := newTracer()
if err != nil { panic(err) }
defer collector.Close()
conn, err := grpc.Dial(host_url, grpc.WithInsecure(), grpc.WithUnaryInterceptor(otgrpc.OpenTracingClientInterceptor(tracer, otgrpc.LogPayloads())))
if err != nil { panic(err) }
defer conn.Close()
client := pb.NewCacheServiceClient(conn)
value, err := callGet(key, client)

Server‑side Tracing Example (Go)

The server creates the same tracer, registers an OpenTracingServerInterceptor, and starts a gRPC server. The interceptor extracts the incoming trace context and creates a child span automatically.

listener, err := net.Listen("tcp", host_url)
if err != nil { panic(err) }
tracer, err := newTracer()
if err != nil { panic(err) }
opts := []grpc.ServerOption{grpc.UnaryInterceptor(otgrpc.OpenTracingServerInterceptor(tracer, otgrpc.LogPayloads()))}
srv := grpc.NewServer(opts...)
cs := initCache()
pb.RegisterCacheServiceServer(srv, cs)
if err = srv.Serve(listener); err != nil { panic(err) } else { fmt.Println("server listening on port 5051") }

Trace and Span Model

A trace represents the entire request flow and is identified by a traceID. A trace consists of multiple spans , each covering a single operation and identified by a spanID. Spans form a directed acyclic graph (DAG) with parent‑child relationships as defined in the OpenTracing Semantic Specification (https://opentracing.io/specification/).

Database Tracing

To trace database calls, use a driver that wraps the standard database/sql interface and starts a child span before each query. Open‑source wrappers include instrumentedsql (https://github.com/ExpansiveWorlds/instrumentedsql), luna‑duclos/instrumentedsql, and ocsql (https://github.com/opencensus-integrations/ocsql/blob/master/driver.go).

Example of adding a child span inside a gRPC service method:

func (c *CacheService) Get(ctx context.Context, req *pb.GetReq) (*pb.GetResp, error) {
    if parent := opentracing.SpanFromContext(ctx); parent != nil {
        pctx := parent.Context()
        if tracer := opentracing.GlobalTracer(); tracer != nil {
            mysqlSpan := tracer.StartSpan("db query user", opentracing.ChildOf(pctx))
            defer mysqlSpan.Finish()
            // simulate DB work
            time.Sleep(10 * time.Millisecond)
        }
    }
    key := req.GetKey()
    value := c.storage[key]
    return &pb.GetResp{Value: value}, nil
}

Cross‑process Tracing

Trace context is propagated via HTTP headers (the “carrier”). The OpenTracingClientInterceptor extracts the current span from the Go context, injects its context into outgoing gRPC metadata, and the server interceptor extracts it back, creating a child span automatically. For pure HTTP services, developers must call Inject and Extract manually or use a compatible interceptor.

Key interceptor code (excerpt from otgrpc/client.go):

func OpenTracingClientInterceptor(tracer opentracing.Tracer, optFuncs ...Option) grpc.UnaryClientInterceptor {
    // ...
    ctx = injectSpanContext(ctx, tracer, clientSpan)
    // ...
}

func injectSpanContext(ctx context.Context, tracer opentracing.Tracer, clientSpan opentracing.Span) context.Context {
    md, ok := metadata.FromOutgoingContext(ctx)
    if !ok { md = metadata.New(nil) } else { md = md.Copy() }
    mdWriter := metadataReaderWriter{md}
    err := tracer.Inject(clientSpan.Context(), opentracing.HTTPHeaders, mdWriter)
    if err != nil { clientSpan.LogFields(log.String("event", "Tracer.Inject() failed"), log.Error(err)) }
    return metadata.NewOutgoingContext(ctx, md)
}

Compatibility Between Tracing Implementations

OpenTracing standardises the API but not the wire format of the trace context. Zipkin uses the X‑B3‑TraceId header, while Jaeger uses uber‑trace‑id. Consequently, a Zipkin‑based service and a Jaeger‑based service cannot interoperate out of the box. Jaeger offers a Zipkin‑compatibility mode that emits B3 headers, enabling cross‑library tracing (https://github.com/jaegertracing/jaeger-client-go/tree/master/zipkin).

Design Recommendations

Cross‑process tracing (service‑to‑service calls) – implement a small interceptor.

Database tracing – use a wrapper driver that automatically creates spans.

In‑process tracing – manually create and finish spans inside functions; this is the most labour‑intensive level.

Service meshes such as Istio or Linkerd can automate cross‑process tracing but still rely on the application to expose trace context for database and in‑process spans.

Trace‑ID Logging

OpenTracing does not expose the trace ID directly. Developers can cast the SpanContext to the concrete implementation (Zipkin or Jaeger) or convert it to a string and parse the ID. A future OpenTracing release will provide a native method for retrieving the trace ID (see https://github.com/opentracing/specification/blob/master/rfc/trace_identifiers.md).

OpenTracing vs. OpenCensus

OpenCensus is a collection of libraries that integrate with various tracing back‑ends, not a generic API like OpenTracing. The two projects are not directly compatible, but they are being merged into a single observability framework (https://medium.com/opentracing/merging-opentracing-and-opencensus-f0fe9c7ca6f0).

Conclusion

Full‑stack tracing in Go microservices consists of generating a correlation ID, propagating it across gRPC or HTTP boundaries, instrumenting database calls with wrapper drivers, and optionally adding manual spans for critical internal functions. Understanding the differences in trace‑context formats between Zipkin and Jaeger is essential for building interoperable observability solutions.

Source Code

Complete example repository: https://github.com/jfeng45/grpcservice