Istio Observability Made Easy: Prometheus, Jaeger & Kiali Guide

Istio Observability

Istio's observability includes metrics, logs, distributed tracing, and visual dashboards. This article explains how to deploy Prometheus for metrics, Jaeger for tracing, and Kiali for visualization.

Prometheus

Configuration Overview

Each Istio component exposes a metrics endpoint that Prometheus scrapes. To collect metrics from the whole mesh, configure Prometheus to scrape the control plane (istiod), ingress/egress gateways, Envoy sidecars, and user applications.

Istio offers two modes to simplify metrics collection:

Option 1: Merge Metrics

Enable the prometheus.io annotations to let Istio add standard scrape annotations to all control‑plane pods. The merged metrics are exposed at /stats/prometheus:15020. This option is enabled by default and can be disabled with --set meshConfig.enablePrometheusMerge=false. When enabled, the data‑plane ports (e.g., 15020) are used.

Use kubectl describe pod to view annotations. Control‑plane ports differ from data‑plane ports (data‑plane: 15020, istiod: 15014, gateways: 15090).

Note that this option exposes all metrics in plaintext and may not suit every scenario (TLS for metrics, name collisions, etc.). You can disable merging per pod with the annotation prometheus.istio.io/merge-metrics: "false".

Option 2: Custom Scrape Configuration

The demo profile installs Prometheus with default scraping. For production, add custom scrape jobs to prometheus/configmap.yaml to collect metrics from control‑plane pods and any sidecars that expose prometheus.io annotations.

TLS Settings

Control‑plane and gateway metrics are exposed in plaintext, but if strict mTLS is enabled for application traffic, Prometheus must use Istio certificates for TLS‑scraping.

Summary of Prometheus Setup

Istio metrics are split into Istio‑generated (plaintext) and application‑generated (subject to TLS). Scraping uses Kubernetes service discovery with prometheus.io/path and prometheus.io/port annotations, which map to meta_kubernetes_pod_annotation_prometheus_io_scrape and meta_kubernetes_pod_annotation_prometheus_io_path in Prometheus.

- job_name: 'kubernetes-pods'
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name

Jaeger

Overview

Distributed tracing lets you follow a request across multiple services, showing latency and visualizing spans. Istio integrates with Envoy’s tracing and supports back‑ends such as Zipkin, Jaeger, and Lightstep.

Trace Context Propagation

Applications must forward headers like x-request-id, x-b3-traceid, x-b3-spanid, x-b3-parentspanid, x-b3-sampled, x-b3-flags, and x-ot-span-context. OpenCensus‑based tracing also requires x-cloud-trace-context, traceparent, and grpc-trace-bin. Example Python code extracts these headers using OpenTracing.

def getForwardHeaders(request):
    headers = {}
    # x-b3-*** headers can be populated using the opentracing span
    span = get_current_span()
    carrier = {}
    tracer.inject(span_context=span.context, format=Format.HTTP_HEADERS, carrier=carrier)
    headers.update(carrier)
    incoming_headers = ['x-request-id', 'x-datadog-trace-id', 'x-datadog-parent-id', 'x-datadog-sampled']
    for ihdr in incoming_headers:
        val = request.headers.get(ihdr)
        if val is not None:
            headers[ihdr] = val
    return headers

Deploying Jaeger

Apply the demo Jaeger manifest:

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.7/samples/addons/jaeger.yaml

Adjust the sampling rate if needed.

Accessing Jaeger

The Jaeger service is named tracing in the istio-system namespace, exposing port 16686. Create an OpenShift route to access the UI.

Generating Traces with Bookinfo

Visit http://$GATEWAY_URL/productpage repeatedly to generate spans. The default sampling rate is 1% (≈100 requests for the first trace).

Use the Jaeger UI to find traces for productpage.default and view span details.

Kiali

Kiali visualizes the entire Istio mesh, showing service graphs, metrics, and configuration validation.

Installation

Deploy Prometheus (required by Kiali):

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.7/samples/addons/prometheus.yaml

Deploy Kiali:

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.7/samples/addons/kiali.yaml

Update custom_metrics_url and url in the Kiali manifest to point to the Prometheus service.

Generating Service Graphs

After deployment, access the Kiali UI, select the bookinfo namespace, and explore different graph types (App, Versioned App, Workload, Service) and edge displays (Response Time, Request Percentage).

Weighted Routing via Kiali

Use the weighted‑routing wizard to split traffic among reviews-v1, reviews-v2, and reviews-v3. Kiali creates a VirtualService and a DestinationRule with the specified weights.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: reviews
  namespace: default
spec:
  hosts:
  - reviews.default.svc.cluster.local
  http:
  - route:
    - destination:
        host: reviews.default.svc.cluster.local
        subset: v1
      weight: 30
    - destination:
        host: reviews.default.svc.cluster.local
        subset: v2
      weight: 0
    - destination:
        host: reviews.default.svc.cluster.local
        subset: v3
      weight: 70
---
kind: DestinationRule
metadata:
  name: reviews
  namespace: default
spec:
  host: reviews.default.svc.cluster.local
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
  - name: v3
    labels:
      version: v3

Configuration Validation

Kiali validates Istio resources and flags errors or warnings. For example, changing a service port name from http to foo triggers an error icon, which can be inspected in the UI.

YAML Editing

Kiali provides a YAML editor with validation highlights. You can view and edit destination rules, virtual services, and other Istio configs directly from the UI.

Kiali Developer API

The API returns JSON representations of graphs and metrics, e.g.,

$KIALI_URL/api/namespaces/graph?namespaces=default&graphType=app

. Use proper app and version labels on workloads for accurate graphs.

Uninstall

kubectl delete -f https://raw.githubusercontent.com/istio/istio/release-1.7/samples/addons/kiali.yaml