Mastering Canary Deployments with Ingress-Nginx Annotations in Kubernetes

Background

In some cases, when using Kubernetes as a cloud platform for business applications, we want to achieve blue‑green deployments for version iteration. Istio is too heavy, while Ingress‑Nginx 0.21 introduced a Canary feature that allows configuring multiple versions of an application at the gateway entry using annotations to control traffic distribution.

Ingress‑Nginx Annotation Canary Feature

To enable the Canary feature, set nginx.ingress.kubernetes.io/canary: "true" and then use the following annotations to configure traffic splitting: nginx.ingress.kubernetes.io/canary-weight – percentage of requests routed to the Canary service (0‑100). nginx.ingress.kubernetes.io/canary-by-header – split traffic based on a request header, useful for gray releases or A/B testing. Values “always” route all traffic to the Canary, “never” route none, other values are ignored. nginx.ingress.kubernetes.io/canary-by-header-value – used together with the header annotation; when the header key and value match, traffic goes to the Canary. nginx.ingress.kubernetes.io/canary-by-cookie – split traffic based on a cookie, with similar “always” and “never” semantics.

Canary rules are evaluated in the order: canary-by-header → canary-by-cookie → canary-weight.

1. Small‑scale version testing based on weight

Define a v1 Ingress and Service:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  labels:
    app: echoserverv1
  name: echoserverv1
  namespace: echoserver
spec:
  rules:
  - host: echo.chulinx.com
    http:
      paths:
      - backend:
          serviceName: echoserverv1
          servicePort: 8080
        path: /
---
kind: Service
apiVersion: v1
metadata:
  name: echoserverv1
  namespace: echoserver
spec:
  selector:
    name: echoserverv1
  type: ClusterIP
  ports:
  - name: echoserverv1
    port: 8080
    targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: echoserverv1
  namespace: echoserver
  labels:
    name: echoserverv1
spec:
  template:
    metadata:
      labels:
        name: echoserverv1
    spec:
      containers:
      - image: mirrorgooglecontainers/echoserver:1.10
        name: echoserverv1
        ports:
        - containerPort: 8080
          name: echoserverv1

After creating the resources, all requests go to the v1 pod.

$ kubectl get pod,service,ingress -n echoserver
NAME                                            READY   STATUS    RESTARTS   AGE
pod/echoserverv1-657b966cb5-7grqs               1/1     Running   0          24h

NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/echoserverv1    ClusterIP   10.99.68.72     <none>        8080/TCP  24h

NAME                                 HOSTS               ADDRESS   PORTS   AGE
ingress.extensions/echoserverv1     echo.chulinx.com    80        24h

Create a v2 service and enable Canary with a weight of 50%:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "50"
  labels:
    app: echoserverv2
  name: echoserverv2
  namespace: echoserver
spec:
  rules:
  - host: echo.chulinx.com
    http:
      paths:
      - backend:
          serviceName: echoserverv2
          servicePort: 8080
        path: /
---
kind: Service
apiVersion: v1
metadata:
  name: echoserverv2
  namespace: echoserver
spec:
  selector:
    name: echoserverv2
  type: ClusterIP
  ports:
  - name: echoserverv2
    port: 8080
    targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: echoserverv2
  namespace: echoserver
  labels:
    name: echoserverv2
spec:
  template:
    metadata:
      labels:
        name: echoserverv2
    spec:
      containers:
      - image: mirrorgooglecontainers/echoserver:1.10
        name: echoserverv2
        ports:
        - containerPort: 8080
          name: echoserverv2

Testing shows roughly half of the requests are routed to v2, demonstrating weight‑based traffic splitting.

2. Header‑based A/B testing

Modify the v2 Ingress to add nginx.ingress.kubernetes.io/canary-by-header: "v2". When the request header is v2:always, all traffic goes to v2; with v2:never, all traffic goes to v1; other values follow the configured weight.

$ kubectl apply -f appv2.yml
ingress.extensions/echoserverv2 configured
service/echoserverv2 unchanged
deployment.extensions/echoserverv2 unchanged

$ for i in `seq 10`; do curl -s -H "v2:always" echo.chulinx.com|grep Hostname; done
Hostname: echoserverv2-856bb5758-f9tqn
Hostname: echoserverv2-856bb5758-f9tqn
... (repeated 10 times)

$ for i in `seq 10`; do curl -s -H "v2:never" echo.chulinx.com|grep Hostname; done
Hostname: echoserverv1-657b966cb5-7grqs
... (repeated 10 times)

$ for i in `seq 10`; do curl -s -H "v2:true" echo.chulinx.com|grep Hostname; done
Hostname: echoserverv1-657b966cb5-7grqs
Hostname: echoserverv2-856bb5758-f9tqn
... (mixed according to weight)

Using nginx.ingress.kubernetes.io/canary-by-header-value allows routing only when the header value matches the specified value.

Cookie‑based routing works analogously, with always and never values controlling traffic distribution.

Summary

Canary releases ensure overall system stability by allowing incremental testing of new versions. The article demonstrates practical use of Ingress‑Nginx annotations to achieve blue‑green, canary, and A/B testing deployments.

Other

About blue‑green deployment, canary release, and A/B testing

Blue‑green deployment uses two identical environments (green – live, blue – new). Traffic is switched from green to blue after validation. Canary release gradually replaces a subset of instances with the new version, controlling traffic via weight. A/B testing runs multiple stable versions simultaneously to compare performance metrics.