Implementing SwimLane Isolation with Service Mesh and Kubernetes: Architecture, WASM Filters, and CRD Controllers

Background

As the company's business scale grows, multiple features are developed and tested concurrently, leading to branch conflicts in a single test environment and reduced development‑test efficiency. In a micro‑service architecture, a feature may depend on several services; changes or failures in those dependencies also affect testing. Moreover, the traditional workflow of code modification → code review → merge to master → redeploy is cumbersome and fragments feature commits.

What is a SwimLane

A SwimLane is an isolated parallel call chain, analogous to lanes in a swimming pool, where calls in different lanes do not interfere with each other. Besides the created lanes, a default "trunk" lane provides the regular test environment. Requests destined for a specific lane are routed only to services deployed in that lane; if a required service is missing, the request falls back to the trunk.

No branch‑occupancy conflicts in the test environment.

Specific versions of dependent services can be pinned to a lane, ensuring stable routing.

Developers can publish to a lane without a full code review, performing an initial test before a final review.

Required Features of a SwimLane

To achieve the above, a SwimLane must provide:

Full‑link forwarding within the lane (unless the lane lacks a service, in which case the trunk is used).

HTTP‑header‑based traffic routing to identify lane‑bound requests.

Correlation of the entire request chain via trace IDs, requiring link tracing.

Lifecycle management for creating, updating, and automatically cleaning up lane resources.

Solution Exploration

The team had already migrated the service architecture to Istio Service Mesh. Istio’s VirtualService and DestinationRule resources enable header‑based routing, but a single‑layer forward is insufficient for full‑link propagation. Full‑link forwarding requires associating outbound requests with the inbound trace ID, which is delegated to the tracing system.

When a request enters a lane, the sidecar stores a mapping of trace_id → label_headers in shared data. Upon exit, the sidecar retrieves the mapping and injects the lane header into the outbound request, achieving end‑to‑end lane propagation. To avoid memory leaks in the default proxy‑wasm shared‑data implementation, a custom shared_cache library is used, which buckets entries and periodically expires them.

Traffic Labelling Envoy Filter

The solution uses WASM extensions (written in Rust) inserted into the Envoy sidecar. Two plugins, traffic_label_inbound and traffic_label_outbound , handle inbound and outbound processing respectively.

Inbound

impl wasm::traits::HttpContext for TrafficLabelInbound {
    fn on_http_request_headers(&mut self, _num_headers: usize, _end_of_stream: bool) -> Action {
        // Remove expired mappings (default 30s)
        self.remove_expired_label_mappings();
        // Extract trace id from request header
        let trace_id = self.get_http_request_header(TRACE_ID_HEADER_NAME);
        if trace_id.is_none() {
            debug!("No trace id header presents in request.");
            return Action::Continue;
        }
        let trace_id = trace_id.unwrap();
        // Collect configured label headers
        let mut label_headers = Headers::new();
        REQUEST_LABEL_HEADERS.with(|headers| {
            if let Some(some_headers) = headers.borrow().deref() {
                for header in some_headers.iter() {
                    let header_value = self.get_http_request_header(header);
                    match header_value {
                        None => { debug!("No label header {} presents in request.", header); continue; }
                        Some(header_value) => {
                            label_headers.push(Header::new(header.clone(), header_value));
                        }
                    }
                }
            }
        });
        if label_headers.is_empty() { return Action::Continue; }
        // Serialize and store in shared cache
        let headers_bytes = bincode::serialize(&label_headers).unwrap();
        SHARED_CACHE.with(|cache| {
            match cache.borrow_mut().set_shared_cache(&trace_id, &headers_bytes[..], None) {
                Ok(_) => debug!("Stored label headers: {:?} for trace: {}", &label_headers, &trace_id),
                Err(e) => error!("Error occurred storing label headers: {:?} for trace: {}, status: {:?}", &label_headers, &trace_id, e),
            }
        });
        // Add trace id to sliding bucket for expiration
        SLIDING_BUCKETS.with(|b| {
            b.borrow_mut().as_mut().unwrap().put(trace_id);
            debug!("Sliding buckets: {:?}", b.borrow());
        });
        wasm::types::Action::Continue
    }
}

Outbound

impl wasm::traits::HttpContext for TrafficLabelOutbound {
    fn on_http_request_headers(&mut self, _num_headers: usize, _end_of_stream: bool) -> Action {
        LABEL_ALL_OUTBOUND_TRAFFIC.with(|label_all_outbound_traffic| {
            if *label_all_outbound_traffic.borrow() {
                LABEL_HEADERS.with(|headers| {
                    let borrowed_headers = headers.borrow();
                    match borrowed_headers.deref() {
                        None => { debug!("No label headers specified for outbound traffic."); return Action::Continue; }
                        Some(headers) => {
                            for header in headers.iter() {
                                VAR_RESOLVER.with(|resolver| {
                                    match resolver.borrow_mut().resolve(&header.value) {
                                        Some(header_value) => {
                                            self.set_http_request_header(&header.name, Some(header_value));
                                            debug!("Added header: {} to request.", &header.name);
                                        }
                                        None => { debug!("No value resolved for header: {:?}.", &header); }
                                    }
                                });
                            }
                            return Action::Continue;
                        }
                    }
                });
            }
        });
        let trace_id = self.get_http_request_header(TRACE_ID_HEADER_NAME);
        if trace_id.is_none() { debug!("No trace id header presents in request."); return Action::Continue; }
        let trace_id = trace_id.unwrap();
        SHARED_CACHE.with(|cache| match cache.borrow().get_shared_cache(&trace_id) {
            Ok(opt_bytes) => match opt_bytes {
                (Some(bytes), _) => {
                    let headers: Result
> = bincode::deserialize(&bytes);
                    match headers {
                        Ok(headers) => {
                            for header in headers.iter() {
                                self.set_http_request_header(&header.name, Some(&header.value));
                                debug!("Added header: {:?} to request with trace id: {}", header, trace_id);
                            }
                        }
                        Err(e) => { error!("Error occurred during deserialization: {}", e); }
                    }
                }
                (None, _) => { debug!("No label headers found with trace id: {}", trace_id); }
            },
            Err(status) => { error!("Unexpected status: {:?}", status); }
        });
        Action::Continue
    }
}

Both filters are packaged into a single WASM module and injected via Istio EnvoyFilter resources for inbound and outbound sidecar listeners.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: traffic-label-wasm-extension
  namespace: istio-system
spec:
  configPatches:
    - applyTo: EXTENSION_CONFIG
      patch:
        operation: ADD
        value:
          name: traffic-label-inbound
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
            config:
              configuration:
                "@type": type.googleapis.com/google.protobuf.StringValue
                value: '{"plugin_type": "traffic_label_inbound", "plugin_conf": "{\"request_label_headers\":[\"fintopia-swim-lane-id\"],\"response_label_headers\":[{\"name\":\"fintopia-swim-lane-id\",\"value\":\"${VERSION:unknown}\"}],\"bucket_time_range_in_secs\":1,\"data_expire_time_in_secs\":30}'}'
              fail_open: true
              name: traffic_label_inbound
              root_id: traffic_label_inbound
              vm_config:
                code:
                  local:
                    filename: /var/local/wasm/dev/traffic_label_all_in_one.wasm
                environment_variables:
                  host_env_keys:
                    - VERSION
                    - LABEL_ALL_OUTBOUND_TRAFFIC
                runtime: envoy.wasm.runtime.v8
                vm_id: traffic_label
    - applyTo: EXTENSION_CONFIG
      patch:
        operation: ADD
        value:
          name: traffic-label-outbound
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
            config:
              configuration:
                "@type": type.googleapis.com/google.protobuf.StringValue
                value: '{"plugin_type": "traffic_label_outbound", "plugin_conf": "{\"label_all_outbound_traffic\":\"${LABEL_ALL_OUTBOUND_TRAFFIC:false}\",\"label_headers\":[{\"name\":\"fintopia-swim-lane-id\",\"value\":\"${VERSION:unknown}\"}]}"}'
              fail_open: true
              name: traffic_label_outbound
              root_id: traffic_label_outbound
              vm_config:
                code:
                  local:
                    filename: /var/local/wasm/dev/traffic_label_all_in_one.wasm
                environment_variables:
                  host_env_keys:
                    - VERSION
                    - LABEL_ALL_OUTBOUND_TRAFFIC
                runtime: envoy.wasm.runtime.v8
                vm_id: traffic_label

SwimLane CRD & Controller

The SwimLane is defined as a custom resource ( SwimLane ) and managed by a controller that reconciles Deployments, VirtualServices, DestinationRules, and optional middleware entries.

Key CRD fields include lane name, namespace, lease period, a list of service deployments (image, name, namespace, feature flags), and middleware specifications.

Controller responsibilities:

Deployment Manager : creates or updates Kubernetes Deployments per lane, adjusts labels, sets replica count to 1, injects environment variables for versioning, Java options, etc.

Informer Service : watches and updates Istio VirtualService and Service objects to ensure traffic is routed based on the lane header.

SubResource Manager : manipulates Istio DestinationRule subsets and VirtualService routing rules to bind a lane’s services to the appropriate version.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: foo-api-destination-rule
  namespace: infra
spec:
  host: foo-api.infra.svc.cluster.local
  subsets:
    - labels:
        version: stable
      name: stable
    - labels:
        version: foo-api-test
      name: foo-api-test

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: foo-api
  namespace: infra
spec:
  hosts:
    - foo-api.infra.svc.cluster.local
  http:
    - match:
        - headers:
            fintopia-swim-lane-id:
              exact: foo-api-test
      name: foo-api-test-swimlane-vs
      route:
        - destination:
            host: foo-api.infra.svc.cluster.local
            subset: foo-api-test
    - name: stable
      route:
        - destination:
            host: foo-api.infra.svc.cluster.local
            subset: stable

Feature Evolution

Java remote debugging (JDWP) integrated into lane deployments.

Middleware lane deployment (e.g., Kafka) via WorkloadEntry and ServiceEntry registration.

Remote hot deployment for rapid code iteration without full rebuild.

For Java debugging, the lane adds the JVM argument -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000 and exposes the port through the gateway, allowing developers to attach IDE debuggers directly to the lane instance.

Middleware deployment registers external services as Istio WorkloadEntry and ServiceEntry objects, enabling lane‑scoped access to resources such as Kafka clusters.

Hot deployment works similarly to Java remote debugging, pushing code changes to the lane’s sidecar in real time.

Conclusion

SwimLane provides a practical use case for Istio Service Mesh in the company, leveraging flexible routing, sidecar extensibility, and custom CRDs to isolate parallel feature development. The addition of Java remote debugging, hot deployment, and middleware lane support further improves developer productivity and reduces conflicts in multi‑feature development scenarios.

The experience gained also deepens the team’s understanding of Istio extensions and cloud‑native service governance, which will be applied to future service‑mesh‑related initiatives.