Mastering Go Microservice Logging and Tracing with OpenTelemetry: An End‑to‑End Guide

Core Concepts

1. Logging

Logging records discrete events generated during system execution, such as request parameters, business flow steps, error stacks, and debug information.

System did what at a certain time?

2. Tracing

Tracing captures the complete call path of a request from entry to exit, together with latency, dependency relationships, and error nodes.

API → Gateway → Order Service → Payment Service → Redis → MySQL

This request executed step‑by‑step? Where is the slowdown? Where did it fail?

3. TraceID

TraceID acts as a unique identifier that bridges logs and traces, enabling precise correlation.

TraceID
 ├─ Service A Log
 ├─ Service B Log
 ├─ Service C Log
 └─ Full call chain

Logs can be aggregated

Traces can be located

Failures can be reconstructed

Solution Comparison

OpenTelemetry (recommended): CNCF standard, strongest ecosystem, multi‑language support.

SkyWalking Go: integrated APM, “plug‑and‑play”.

GoFrame: built‑in framework support, selected GoFrame.

go-zero: built‑in framework support, selected go-zero.

We adopt OpenTelemetry as the standard solution.

Environment and Dependencies

go get go.opentelemetry.io/otel \
     go.opentelemetry.io/otel/trace \
     go.opentelemetry.io/otel/sdk \
     go.opentelemetry.io/otel/exporters/jaeger \
     go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

Initialize TracerProvider (Production‑grade)

func InitTracer(serviceName string) (*sdktrace.TracerProvider, error) {
    exp, err := jaeger.New(
        jaeger.WithCollectorEndpoint(
            jaeger.WithEndpoint("http://jaeger-collector:14268/api/traces"),
        ),
    )
    if err != nil {
        return nil, err
    }

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exp),
        sdktrace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName(serviceName),
        )),
        sdktrace.WithSampler(
            sdktrace.ParentBased(
                sdktrace.TraceIDRatioBased(0.01), // 1% sampling
            ),
        ),
    )
    otel.SetTracerProvider(tp)
    return tp, nil
}

Automatic Instrumentation (HTTP)

handler := otelhttp.NewHandler(myHandler, "http-server")
http.ListenAndServe(":8080", handler)

Request Span

HTTP Method / URL

Status code

Latency

TraceID injected into Context

Returning TraceID to Client

func TraceMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        sc := trace.SpanFromContext(r.Context()).SpanContext()
        if sc.IsValid() {
            w.Header().Set("X-Trace-Id", sc.TraceID().String())
        }
        next.ServeHTTP(w, r)
    })
}

Complete loop: client → X‑Trace‑Id → query logs → query trace → pinpoint issue.

客户端 → X-Trace-Id → 查日志 → 查链路 → 精确定位问题

Business‑level Span Creation

func businessHandler(ctx context.Context) {
    tracer := otel.Tracer("order-service")
    ctx, span := tracer.Start(ctx, "CreateOrder")
    defer span.End()

    span.SetAttributes(
        attribute.String("order.type", "normal"),
        attribute.Int("user.id", 123),
    )

    doPay(ctx)
}

Logging with TraceID (Production Practice)

Use the Zap logger.

go get go.uber.org/zap

func LoggerWithTrace(ctx context.Context, logger *zap.Logger) *zap.Logger {
    sc := trace.SpanFromContext(ctx).SpanContext()
    if sc.IsValid() {
        return logger.With(
            zap.String("trace_id", sc.TraceID().String()),
            zap.String("span_id", sc.SpanID().String()),
        )
    }
    return logger
}

Usage:

LoggerWithTrace(ctx, logger).
    Info("创建订单成功", zap.Int("order_id", 1001))

Resulting log structure:

{
  "time": "2026-01-26T10:00:00Z",
  "level": "INFO",
  "trace_id": "8f2c...",
  "span_id": "2a1b...",
  "service": "order-service",
  "msg": "创建订单成功",
  "order_id": 1001
}

Sampling Strategy Design (Must)

sdktrace.WithSampler(
    sdktrace.ParentBased(
        sdktrace.TraceIDRatioBased(0.01),
    ),
)

Core transactions: 5% sampling

Normal APIs: 1% sampling

Auxiliary systems: 0.1% sampling

Log everything, trace sampled.

Production‑grade Architecture

[Service A]   [Service B]   [Service C]
     |            |            |
     +-------- OpenTelemetry SDK --------+
                 |
              OTEL Collector
                 |
   +-------------------+-------------------+
   |                   |                   |
Jaeger/Tempo      Loki/ES               Prometheus
(Tracing)         (Logging)            (Metrics)

The collector receives data, performs sampling, transforms formats, and forwards to back‑ends.

Relationship Summary

Tracing – system “timeline”.

Logging – annotations on the timeline.

Metrics – system “vital signs”.

Example trace structure:

Trace
 ├─ Span A
 │   ├─ log1
 │   ├─ log2
 ├─ Span B
      ├─ log3

Final Takeaway

In Go microservices, tracing is the skeleton, logging is the flesh, and metrics are the heartbeat. Without a TraceID, logs cannot be globally correlated; without logs, a trace cannot explain business details. Combining OpenTelemetry, TraceID‑bound logs, and visual back‑ends yields an industrial‑grade, auditable, traceable observability system.