From Push to Production: Mastering Quality Gates, Canary Releases, and Rollback Strategies

Overview: Release System = Automation + Quality Gates + Rollback

The author frames CI/CD not as mere automation but as a "survival system" for releases. Automation alone pushes bad code to production; gates alone lack incident response. The complete system combines automation, quality gates (blocking unqualified versions), and rollback/stop-loss (fast recovery when faults occur).

Seven-Stage Pipeline with Gates and Rollback

Each stage defines inputs, quality gates, and rollback actions:

Commit : Action = Code push + MR; Output = commit sha; Quality Gate = Branch protection rules; Rollback = N/A

Validate : Action = Static scan + unit tests; Output = Test report, coverage; Quality Gate = Unit test failure blocks merge; Rollback = Reject MR

Build : Action = Compile + build image; Output = Image digest; Quality Gate = Build failure blocks; Rollback = Discard build

Push : Action = Push to ECR; Output = Immutable image; Quality Gate = Critical vulnerability blocks release; Rollback = Delete image tag

Approve : Action = Release review; Output = Release ticket; Quality Gate = No deploy without approval; Rollback = Cancel ticket

Deploy : Action = K8s rolling update; Output = Ready pods; Quality Gate = Readiness probe + smoke test; Rollback = Rollback to previous digest

Canary : Action = Weighted traffic split; Output = Traffic ratio; Quality Gate = Metric anomaly triggers auto-rollback; Rollback = ALB weight to 0

Key insight: the pipeline's core artifact is the Docker image (identified by immutable digest), not the JAR. Tags can be overwritten; digest never changes, enabling traceability, rollback, and canary routing.

Testing Layers: Unit, Integration, Smoke, Regression

Unit : Granularity = Method/class; Dependencies = All mocked; When = Build phase; Failure Means = Business logic error

Integration : Granularity = Multi-component; Dependencies = Real middleware; When = Post-build; Failure Means = Contract incompatibility

Smoke : Granularity = End-to-end; Dependencies = Production env; When = Post-deploy; Failure Means = Startup, config, downstream issues

Regression : Granularity = Full chain; Dependencies = Production env; When = Pre/post release; Failure Means = Hidden side-effects, compat bugs

Unit test example (Mockito):

@Test
void 库存不足时抛出异常() {
    when(stockMapper.decr(anyLong(), anyInt())).thenReturn(0);
    assertThrows(StockNotEnoughException.class,
        () -> seckillService.deduct(1001L, 2));
}

Gate policies: overall line coverage < 60% blocks MR merge; new code coverage < 80% warns only; flaky tests auto-retry once, then quarantined for refactor. Legacy code not forced to coverage to avoid pipeline gridlock.

Event-Driven Pipeline Trigger with HMAC Verification

Webhook (not polling) is production standard. Git sends HMAC-signed payload to Jenkins; Jenkins verifies with shared secret:

def checkWebhookSecret(String secret, String headerSign) {
    return hmacVerify(secret, env.GIT_BODY, headerSign)
}

Mismatch drops request, preventing malicious triggers.

K8s Deployment: ECR Pull via Secret + ServiceAccount

containerd (not Docker) pulls images. kubelet drives containerd using credentials from a Secret bound to a ServiceAccount:

apiVersion: v1
kind: Secret
metadata:
  name: ecr-pull-secret
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: <base64 docker config>

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-sa
imagePullSecrets:
- name: ecr-pull-secret

Pull flow: kubelet gets creds → OCI request manifest → ECR issues temp token → pull layer blobs → local unpack & start. Common ImagePullBackOff causes: 401 (expired creds), 403 (IAM/perms), timeout (network/DNS). Debug with crictl pull on node. Advanced: node IAM Role + credential provider for auto-refreshing tokens.

Post-Deploy Verification: Probe, Smoke, Canary

Readiness vs Liveness : readiness controls traffic reception (slow startup, downstream not ready → pod removed from Service); liveness detects process deadlock (only then restart). Mixing causes unnecessary churn.

readinessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 10

Smoke test runs after readiness: login, product query, pre-deduct inventory. All pass → enter canary; any fail → auto-rollback. Implemented via external test platform:

curl -X POST https://test-platform/smoke/run 
  -d '{"service":"order","env":"prod"}'

Canary strategies (decoupled routing & observability): 1. Replica ratio (coarse, internal services) 2. Nginx Ingress Canary (Ingress-layer weight) 3. ALB dual target groups (primary, stable, good observability) 4. Istio VirtualService (header/user-ID routing, fine-grained) Version identified by image digest, injected request header, and version dimension in logs/metrics — not URL path.

Database Migration: Expand-Contract Pattern

Goal: new/old code coexist reading/writing same table; DDL decoupled from app deploy.

7.1 Expand (add only)

ALTER TABLE orders ADD COLUMN settle_channel VARCHAR(32) DEFAULT NULL;
CREATE INDEX CONCURRENTLY idx_orders_settle_channel ON orders(settle_channel);

Old code unaffected; new column nullable.

7.2 Dual-write

orderMapper.insert(order);
orderExtMapper.insertSettleChannel(order.getId(), channel);

Rollback to old version loses no data.

7.3 Cutover via feature flag

if (featureSwitch.readNewSettleColumn()) {
    return orderExtMapper.getSettleChannel(orderId);
} else {
    return order.getLegacyChannel();
}

Avoid during canary: rename column, drop column, add non-null no-default, change type/charset. Standard cadence: N-1 run DDL; N deploy dual-write + backfill; N+1 cut read, stop dual-write, wait one cycle, drop old column.

Rollback/Stop-Loss: 30-Second Target

Core principle: reuse existing image (no rebuild). - Image digest unchanged → roll back to previous stable digest - ALB weight to 0 → traffic cuts to old version in seconds - Feature flag off → DB compatible, no DDL rollback needed - Expand-contract ensures code rollback never hits missing-column errors

Canary monitors: 5xx/4xx rate, P99/P999 latency, business metrics (order success, inventory pre-deduct, payment callback). Alert rules compare new vs old version; breach triggers auto-rollback without human intervention.

Five-Layer Defense-in-Depth

Commit layer : unit tests + static scan block

Image layer : vulnerability scan block

Digest layer : readiness probe + smoke test block

Weight layer : canary metric anomaly block

Switch layer : DB compatibility fallback (feature flags + expand-contract)

Missing any layer lets faults propagate. CI/CD's essence is controlled change, not speed. Automation accelerates; gates block risk; rollback bounds impact. Goal: not zero faults, but minimized blast radius and recovery time — every change stays within controllable boundaries.