Zero‑Code Traffic Lane Gray Release with OpenTelemetry Baggage on Alibaba Cloud Service Mesh

Overview

The article introduces traffic lanes as isolated runtime environments for different service versions within a cloud‑native application. In strict mode, each lane contains the full call chain, while loose mode requires only a baseline lane and allows other lanes to omit some services, relying on a propagated request header (baggage) for routing.

OpenTelemetry Baggage

OpenTelemetry defines a baggage request header that carries key‑value pairs across the distributed call chain. By enabling the baggage propagator in the OpenTelemetry Operator, services can automatically forward this header without code changes.

Prerequisites

Alibaba Cloud Service Mesh (ASM) Enterprise or Premium instance version ≥ 1.21.6.54.

Cluster added to the ASM instance.

Ingress gateway named ingressgateway created.

Gateway rule for ingressgateway configured.

Step 1 – Deploy OpenTelemetry Operator

Create the opentelemetry-operator-system namespace and install the operator with Helm:

kubectl create namespace opentelemetry-operator-system</code>
<code>helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts</code>
<code>helm install \
    --namespace=opentelemetry-operator-system \
    --version=0.46.0 \
    --set admissionWebhooks.certManager.enabled=false \
    --set admissionWebhooks.certManager.autoGenerateCert=true \
    --set manager.image.repository="registry-cn-hangzhou.ack.aliyuncs.com/acs/opentelemetry-operator" \
    --set manager.image.tag="0.92.1" \
    --set kubeRBACProxy.image.repository="registry-cn-hangzhou.ack.aliyuncs.com/acs/kube-rbac-proxy" \
    --set kubeRBACProxy.image.tag="v0.13.1" \
    --set manager.collectorImage.repository="registry-cn-hangzhou.ack.aliyuncs.com/acs/opentelemetry-collector" \
    --set manager.collectorImage.tag="0.97.0" \
    --set manager.opampBridgeImage.repository="registry-cn-hangzhou.ack.aliyuncs.com/acs/operator-opamp-bridge" \
    --set manager.opampBridgeImage.tag="0.97.0" \
    --set manager.targetAllocatorImage.repository="registry-cn-hangzhou.ack.aliyuncs.com/acs/target-allocator" \
    --set manager.targetAllocatorImage.tag="0.97.0" \
    --set manager.autoInstrumentationImage.java.repository="registry-cn-hangzhou.ack.aliyuncs.com/acs/autoinstrumentation-java" \
    --set manager.autoInstrumentationImage.java.tag="1.32.1" \
    --set manager.autoInstrumentationImage.nodejs.repository="registry-cn-hangzhou.ack.aliyuncs.com/acs/autoinstrumentation-nodejs" \
    --set manager.autoInstrumentationImage.nodejs.tag="0.49.1" \
    --set manager.autoInstrumentationImage.python.repository="registry-cn-hangzhou.ack.aliyuncs.com/acs/autoinstrumentation-python" \
    --set manager.autoInstrumentationImage.python.tag="0.44b0" \
    --set manager.autoInstrumentationImage.dotnet.repository="registry-cn-hangzhou.ack.aliyuncs.com/acs/autoinstrumentation-dotnet" \
    --set manager.autoInstrumentationImage.dotnet.tag="1.2.0" \
    --set manager.autoInstrumentationImage.go.repository="registry-cn-hangzhou.ack.aliyuncs.com/acs/opentelemetry-go-instrumentation" \
    --set manager.autoInstrumentationImage.go.tag="v0.10.1.alpha-2-aliyun" \
    opentelemetry-operator open-telemetry/opentelemetry-operator

Verify the operator pods are running:

kubectl get pod -n opentelemetry-operator-system

Step 2 – Deploy Example Services

Create an Instrumentation resource that enables the baggage propagator:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: demo-instrumentation
spec:
  propagators:
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "1"

Apply it: kubectl apply -f instrumentation.yaml Deploy three mock services (mocka, mockb, mockc) with three versions (v1, v2, v3). Each deployment includes the annotation instrumentation.opentelemetry.io/inject-java: "true" and instrumentation.opentelemetry.io/container-names: "default" so the operator injects the Java auto‑instrumentation sidecar.

# Example snippet for mocka‑v1
apiVersion: v1
kind: Service
metadata:
  name: mocka
  labels:
    app: mocka
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mocka
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mocka-v1
  labels:
    app: mocka
    version: v1
    ASM_TRAFFIC_TAG: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mocka
      version: v1
      ASM_TRAFFIC_TAG: v1
  template:
    metadata:
      labels:
        app: mocka
        version: v1
        ASM_TRAFFIC_TAG: v1
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        env:
        - name: version
          value: v1
        - name: app
          value: mocka
        - name: upstream_url
          value: "http://mockb:8000/"
        ports:
        - containerPort: 8000

Repeat similarly for mockb and mockc, and for versions v2 and v3.

Step 3 – Create Lane Group and Lanes

In the ASM console, create a lane group named test with the ingress gateway ingressgateway and select Loose mode . Set the traffic‑context propagation method to baggage and specify the header x-asm-prefer-tag.

Add services mocka, mockb, mockc to the group.

Create three lanes (s1, s2, s3) and bind them to the tags v1, v2, v3 respectively:

s1: mocka, mockb, mockc (all default namespace)

s2: mocka, mockc

s3: mockb

ASM automatically generates DestinationRule and VirtualService resources for each service. Example for mocka:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  labels:
    asm-system: 'true'
    provider: asm
    swimlane-group: test
  name: trafficlabel-dr-test-default-mocka
  namespace: istio-system
spec:
  host: mocka.default.svc.cluster.local
  subsets:
  - name: s1
    labels:
      ASM_TRAFFIC_TAG: v1
  - name: s2
    labels:
      ASM_TRAFFIC_TAG: v2

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  labels:
    asm-system: 'true'
    provider: asm
    swimlane-group: test
  name: trafficlabel-vs-test-default-mocka
  namespace: istio-system
spec:
  hosts:
  - mocka.default.svc.cluster.local
  http:
  - match:
    - headers:
        x-asm-prefer-tag:
          exact: s1
    route:
    - destination:
        host: mocka.default.svc.cluster.local
        subset: s1
      fallback:
        target:
          host: mocka.default.svc.cluster.local
          subset: s1
  - match:
    - headers:
        x-asm-prefer-tag:
          exact: s2
    route:
    - destination:
        host: mocka.default.svc.cluster.local
        subset: s2
      fallback:
        target:
          host: mocka.default.svc.cluster.local
          subset: s1
  - match:
    - headers:
        x-asm-prefer-tag:
          exact: s3
    route:
    - destination:
        host: mocka.default.svc.cluster.local
        subset: s3
      fallback:
        target:
          host: mocka.default.svc.cluster.local
          subset: s1

Step 4 – Define Weight‑Based Unified Routing

In the lane group, create a weight‑based routing rule for the entry path /mock. Set the domain to * and match the URI prefix /. Assign traffic weights: s1 = 60, s2 = 20, s3 = 20.

Step 5 – Verify Gray‑Release Behavior

Obtain the public IP of the ASM ingress gateway and export it as ASM_GATEWAY_IP: export ASM_GATEWAY_IP=xxx.xxx.xxx.xxx Send repeated requests and observe the service chain output. Expected output shows roughly a 6:2:2 distribution among the three lanes, with lane s1 acting as the baseline when a target service version is missing.

-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)
... (other lines showing v2 and v3 according to the configured weights) ...

Conclusion

The guide demonstrates a non‑intrusive loose‑mode traffic‑lane implementation using OpenTelemetry’s baggage propagation, enabling fine‑grained gray releases in Alibaba Cloud Service Mesh without modifying application code, and shows how weight‑based routing can distribute traffic across multiple lanes.