How to Enable and Analyze Istio Distributed Tracing with Jaeger on Kubernetes

Background & Introduction

Distributed tracing records and correlates details of each stage in a distributed call, helping operators quickly locate failures or performance bottlenecks. Istio uses Jaeger, an open‑source tracing system from Uber, as its tracing backend for storage, visualization, and querying.

Special Notes & Constraints

Although Istio claims to be non‑intrusive, tracing still requires intercepting and forwarding request headers. The official documentation explains that the Istio sidecar must add appropriate HTTP headers so that spans can be linked into a single trace.

Despite the Istio proxy automatically sending spans, additional information is needed to attach these spans to the same trace. Each application must collect request headers from incoming requests and forward them to all outbound requests. The specific headers to forward depend on the configured tracing backend.

Key request headers include:

x-request-id – Envoy‑specific unique identifier for logs and tracing.

x-b3-traceid – Trace ID generated by the first span and propagated downstream.

x-b3-spanid – Span ID assigned when a span is created.

x-b3-parentspanid – Parent span ID for the current call.

x-b3-sampled – Sampling flag (1 to report the span, 0 to ignore).

x-b3-flags – Additional flags.

These headers allow the server to reconstruct the trace after all span data is pushed.

How Istio‑Proxy Generates Span Information

For inbound traffic, if no tracing headers are present, Envoy creates a root span and injects its context into the request. If tracing headers exist, Envoy extracts the context and forwards it to the application. For outbound traffic, Envoy creates a root span when no headers are present, or creates a child span based on the received context and adds the new span information to the outbound request headers.

Note: This mechanism requires HTTP communication between services so that headers can be propagated.

Deploy Jaeger & Test Demo

Prepare a Kubernetes cluster with Istio installed (e.g., Kubernetes 1.25 and Istio 1.19). Deploy Jaeger manually because Istio does not enable it by default.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger
  namespace: istio-system
  labels:
    app: jaeger
spec:
  selector:
    matchLabels:
      app: jaeger
  template:
    metadata:
      labels:
        app: jaeger
        sidecar.istio.io/inject: "false"
    spec:
      containers:
        - name: jaeger
          image: "docker.io/jaegertracing/all-in-one:1.46"
          env:
            - name: BADGER_EPHEMERAL
              value: "false"
            - name: SPAN_STORAGE_TYPE
              value: "badger"
            - name: BADGER_DIRECTORY_VALUE
              value: "/badger/data"
            - name: BADGER_DIRECTORY_KEY
              value: "/badger/key"
            - name: COLLECTOR_ZIPKIN_HOST_PORT
              value: ":9411"
            - name: MEMORY_MAX_TRACES
              value: "50000"
            - name: QUERY_BASE_PATH
              value: /jaeger
          livenessProbe:
            httpGet:
              path: /
              port: 14269
          readinessProbe:
            httpGet:
              path: /
              port: 14269
          volumeMounts:
            - name: data
              mountPath: /badger
          resources:
            requests:
              cpu: 10m
      volumes:
        - name: data
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: tracing
  namespace: istio-system
  labels:
    app: jaeger
spec:
  type: ClusterIP
  ports:
    - name: http-query
      port: 80
      protocol: TCP
      targetPort: 16686
    - name: grpc-query
      port: 16685
      protocol: TCP
      targetPort: 16685
  selector:
    app: jaeger
---
# Additional services for Zipkin compatibility and collector ports omitted for brevity

Deployment succeeded:

Adjust Sampling Rate

Istio’s default sampling rate is 1 %. Increase it to 100 % for full visibility.

kubectl edit iop installed-state -n istio-system

spec:
  values:
    pilot:
      traceSampling: 100  # default 1, change to 100

Deploy Bookinfo Demo

Deploy the official Bookinfo microservices (details, ratings, reviews, productpage) using the provided Kubernetes manifests.

# Sample excerpt of the Bookinfo manifests (full YAML omitted for brevity)
apiVersion: v1
kind: Service
metadata:
  name: details
  labels:
    app: details
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: details
---
# Additional services and deployments for ratings, reviews, and productpage follow the same pattern

Deployment succeeded:

Testing & Observations

Access the productpage service via its ClusterIP and trigger a request. The request succeeds, and Jaeger’s UI shows generated spans: 7 spans across 4 services with a depth of 5 layers.

Analyzing the Trace

Set the Bookinfo pods’ log level to trace to capture detailed span information.

spec:
  template:
    metadata:
      annotations:
        sidecar.istio.io/logLevel: trace

Log analysis shows that productpage generates three spans (one inbound, two outbound), reviews generates two spans, and each of ratings and details generates one inbound span, matching the seven spans displayed in Jaeger.

'x-b3-traceid': '235aa67f7b07cc91bbdd05f23bcd7bab', 'x-b3-spanid': 'bbdd05f23bcd7bab'

The trace ID remains consistent across the entire call chain.

Business Code Logic

Tracing is performed by the sidecar; application code only needs to forward the received trace headers.

@app.route('/productpage')
@trace()
def front():
    product_id = 0  # TODO: replace default value
    headers = getForwardHeaders(request)
    ...

def getForwardHeaders(request):
    headers = {}
    span = get_current_span()
    carrier = {}
    tracer.inject(span_context=span.context, format=Format.HTTP_HEADERS, carrier=carrier)
    headers.update(carrier)
    if 'user' in session:
        headers['end-user'] = session['user']
    return headers

def getProductDetails(product_id, headers):
    try:
        url = details['name'] + "/" + details['endpoint'] + "/" + str(product_id)
        res = requests.get(url, headers=headers, timeout=3.0)
        ...

In the Details service, incoming headers are collected and forwarded downstream.

def get_forward_headers(request):
    headers = {}
    incoming_headers = [
        'x-request-id',
        'x-ot-span-context',
        'x-datadog-trace-id',
        'x-datadog-parent-id',
        'x-datadog-sampling-priority',
        'x-b3-traceid',
        'x-b3-spanid',
        'x-b3-parentspanid',
        'x-b3-sampled',
        'x-b3-flags',
        'end-user',
        'user-agent',
        'cookie',
        'authorization',
        'jwt'
    ]
    request.each do |header, value|
        if incoming_headers.include? header then
            headers[header] = value
        end
    end
    return headers
end