Spring Cloud Gray Release: Beyond Gateway Routing to Full-Chain Context Propagation

Three Real Incidents Opening

Incident 1: Gray only at gateway, second hop leaked. Gateway routed 5% traffic by user ID to v2, but v2's Feign call to order service carried no tag, hitting v1 instances. v1 returned JSON missing v2's new field causing NPE; worse, v2 wrote a record with the new field, v1 deserialization threw exception, collapsing the whole query chain. Gray is not "gateway splitting", it's "dyeing through" — where the tag breaks, gray ends.

Incident 2: Configured by instance ratio, actual traffic only 0.3%. 10 instances, 1 gray, expecting 10% traffic. Result: gray instance QPS only single digits. Cause: entire chain is "sticky": HTTP keep-alive, connection pool reuse, load balancer 35s instance list cache — weight 10% but connections already established, requests don't redistribute because new instance appears. Instance ratio ≠ traffic ratio, stickier the chain, worse the skew.

Incident 3: Gray three days normal, full release exploded in one hour. Gray version added a NOT NULL column without default; gray instance executed new DDL. During gray only it wrote, no issue; after full release old instances still running, INSERT without that column errored. Meanwhile cache mixed two serialization formats, reads half null. Gray must verify not "new version runs" but "old and new coexist".

Three incidents, three main threads: Dyeing must penetrate, proportion must be calculated by traffic, compatibility must be bidirectional. The whole article expands around these three — they matter more than any routing component config.

Technical Analysis: First Distinguish Which Layer You're Doing

2.1 Three Levels, Mixing Them Causes Mistakes

① Entrance Splitting
Gateway / Nginx / LB weight
Only decides "which group this request enters"

② Instance Grouping
Nacos metadata / K8s label / independent Deployment
Decides "which instances belong to new version"

③ Full-Chain Dyeing
Tag propagates along call chain
Every hop selects instance by tag

Only the third level is "gray release". Doing only first two gives "some requests hit new version" — works in monoliths (single call), but in microservices almost inevitably causes Incident 1: first hop new, rest old, you think you're validating v2 but actually validating a hybrid. Worse, hybrid metrics are meaningless: v2 error rate mixed with v1 behavior, so "gray looks stable" and "full release immediately fails" can both be true — not gray deceiving you, you measured the wrong thing.

2.2 Four Routing Dimensions, Choose Wrong and Suffer

By Ratio
Simplest, suits stateless read APIs
Drawback: same user may flip between versions

By User/Tenant ID Hash
Same user always same version
Suits ordering, payment continuous flows

By Tag (header/cookie/whitelist)
Precisely specify internal users first
Most used for joint debugging & acceptance

By Business Content (region/device/level)
Business dimension gray, A/B common
Needs extra bucket rule maintenance

Practical mnemonic: If you can stick by user, don't use ratio. An ordering flow where first hop hits v2, second hits v1 shows user "product has discount, checkout no discount" contradictory UI, hardest to reproduce — depends on connection distribution, one retry and it's gone. Ratio only fits truly stateless: product detail, config, static resources. Once chain has "previous writes, next reads" relationship, must guarantee same entity lands on same version.

2.3 Three Iron Laws of Multi-Version Coexistence

Iron Law 1: Interface backward compatible. New version only adds fields, cannot change semantics, delete fields, change types. Old service ignoring new fields is fine (Jackson defaults ignore unknown, unless FAIL_ON_UNKNOWN_PROPERTIES enabled); but if new service depends on old service definitely returning a field , that's a one-way ticket — gray period guaranteed NPE.

Iron Law 2: Database schema only expands, never contracts. Add column (nullable or with default), add table, add index OK; drop column, change type, add NOT NULL, change length — all pushed to "after confirming no old version running". Violating this once is Incident 3.

Iron Law 3: Message format dual-read. Producer upgraded sends new format, consumer must handle both old and new; conversely gray-period old producer still sends old format, new consumer must recognize. So message body always keep a schema version field , consumer branches by version, far more reliable than guessing if field has value.

Why these three are so hard: Gray's essence is "two versions alive simultaneously", not "new replaces old process". As long as coexistence, data layer is shared, and data layer unlike code cannot instantly roll back — code rollback 30 seconds, data rollback may require downtime.

2.4 Why Instance Ratio ≠ Traffic Ratio

Long connections / Connection pools
Connections once built keep used, not reallocated
HTTP/2, gRPC, DB pools all same

LB Instance List Cache
Spring Cloud LoadBalancer 35s
Ribbon era 30s

DNS Cache
JVM default permanent cache resolution
K8s Service changed still not refresh

Session Stickiness / Affinity
sticky session, same instance priority

Traffic Itself Uneven
Big customers, hot keys, scheduled batches

Five factors stacked, "1/10 instances = 10% traffic" holds only in one case: huge request volume, ultra-short connections, rebuilding every second. Almost only public short-connection APIs satisfy. Internal services, actual deviation usually multiples to tens of times. So to precisely control proportion, must decide at entrance per request, then propagate decision downstream ; splitting by instance count only suits "I want a tiny bit of real traffic" vague demand, not "I want exactly 5%".

2.5 Gray, Blue-Green, Canary, A/B, Shadow — Don't Mix Up

Gray/Canary  Small real traffic, verify stability
Goal: Safe release  Cost: Time

Blue-Green    Two full environments, instant switch
Goal: Fast rollback  Cost: Double resources

A/B          Bucket by user, compare business metrics
Goal: Verify effect  Cost: Statistical design

Shadow       Mirror traffic, not returned to user
Goal: Stress verify  Cost: Dual-write risk

Most common mismatch: using A/B framework for gray. A/B cares "is new better", needs sample size and significance; gray cares "will new explode", needs error rate, latency, resources stability metrics. Using A/B mindset for gray falls into "sample insufficient so wait longer" loop, while real watch is P99 rising. Another common: using blue-green to replace gray. Blue-green switch is all-at-once, moment of switch all users enter new version, all prior small-traffic validation gains wiped out. Blue-green solves "rollback fast", not "risk small".

Core Flow: How a Tagged Call Walks

3.1 Where Tag Comes From, Decide Only Once

① Gateway Decides
Whitelist -> UserID Hash -> Random Ratio
Priority top-down, hit stops

② Write into Request Header
X-Gray-Tag: v2
Don't use multiple headers for one thing

③ After That All Propagation
No hop allowed to re-decide

Third rule is discipline, also most easily broken. Common break: "some service adds its own gray judgment": gateway says go v1, service internally rolls its own dice, request jumps between v1 and v2. Decision point one and only one, other nodes only read not judge. If truly need multi-level decision (gateway by user, service by tenant), agree on a "already decided" marker: header carries this marker then fully trust, no second judgment. Otherwise rules fight, nobody can tell which path a request actually took.

3.2 Five Context Propagation Breakpoints

① HTTP Entrance
Filter reads header -> stores in context
② In-Process Call
ThreadLocal or RequestContextHolder
③ Outbound Client
Feign Interceptor / RestTemplate Interceptor
/ WebClient Filter — all three must write
④ Thread Pool Switch
TaskDecorator captures and restores
⑤ Message Queue
Producer stuffs into message header
Consumer extracts and restores

These five breakpoints completely overlap with distributed tracing — not coincidence, TraceId and gray tag are essentially same thing: a context that must follow the request . So easiest implementation reuses same propagation mechanism.

3.3 Instance Selection: Filter by Metadata

Nacos Side
Gray group startup param add
--spring.cloud.nacos.discovery.metadata.version=gray
Production group not set or stable

Consumer Side
Extract tag from header
Filter instance list: metadata.version equals tag
Empty -> fallback to all (mandatory)

Using metadata not "instance IP whitelist" because metadata pushes with registration to all consumers, no need to maintain an IP list in config center . IP list problem: instance rebuild IP changes, list always lags, and hard-codes logical concept "version" into infrastructure detail.

3.4 One Must-Write, Often-Forgotten Fallback

After filtering list empty what to do? Most handwritten defaults: get empty list, take first, throw exception, return 500. When empty? Gray group scaled down, scaling not registered yet, some downstream has no gray version, someone manually changed metadata. All four common in production.

Correct behavior: Fallback to all instances, and emit a warn log + a metric. Reason direct: gray is "safer path", not "only path"; failing requests to force gray turns optimization into failure. But fallback must be observable — if it falls back daily, your gray strategy never effective, just undiscovered.

3.5 Should Gray Group Deploy Independently

Option A Same Deployment with Label
Saves resources, rolling update easily mixes versions
Option B Independent Deployment
Clear version boundary, resources must reserve
Recommended: Gray group independent, fixed replicas

Recommend B. Reason not technical, but accidents: same Deployment mixing two versions, K8s rolling updates at its own rhythm, your "gray 1 instance" may become "rolled half". Troubleshooting Pod list can't tell who's who. Independent Deployment costs extra resource reservation. This cost can be compressed by "gray group resident 1-2 replicas, scale on demand": normally keep just enough for validation, scale two more before big promo. Key is making "gray" an explicit, enumerable object, not scattered if-judgments.

Core Source: How to Plug Into LoadBalancer Filter Chain

4.1 ServiceInstanceListSupplier Is a Responsibility Chain

DiscoveryClientServiceInstanceListSupplier
-> Fetch all instances from registry
CachingServiceInstanceListSupplier
-> Cache 35s (default)
GrayInstanceListSupplier (custom)
-> Filter by tag

Key: Filter MUST be placed AFTER cache

Order matters: if filter before cache, cache stores only gray instances, production traffic reads cache gets no available instances, immediate site-wide 500. Placed after, cache holds full list, each request does one filter — filter is in-memory collection ops, traversing dozens of instances overhead negligible . Another anti-pattern: writing filter in choose(). Looks equivalent, actually loses supplier chain composability — later adding "same zone priority" "weight" logic piles ifs in choose, becomes code nobody dares touch.

4.2 Custom Supplier Skeleton

public class GrayInstanceSupplier
implements ServiceInstanceListSupplier {
private final ServiceInstanceListSupplier d;

public Flux<List<ServiceInstance>> get() {
return d.get().map(list -> {
String tag = GrayContext.get();
if (tag == null) return list;
List<ServiceInstance> hit =
list.stream()
.filter(i -> tag.equals(
    i.getMetadata()
    .get("version")))
.collect(Collectors.toList());
if (hit.isEmpty()) {
    GrayMetrics.fallback();
    return list; // fallback all
}
return hit;
});
}
public String getServiceId() {
return d.getServiceId();
}
}

Two easily missed details. First, GrayContext.get() must be "current request" context, not a static variable — if static, concurrent requests overwrite each other's tag, under load test shows "gray ratio completely random". Second, fallback branch must emit metric, not print. Third detail in assembly: this configuration class must NOT be in @ComponentScan scan range , otherwise becomes global default, all services go through gray filter. Correct way: specify via @LoadBalancerClients, let each client have independent child container. This pit same as Feign configuration in article 03: parent-child container isolation not understood, config pollutes globally .

4.3 Outbound Trio, Miss One and a Path Lacks Tag

// Feign
public class GrayFeignInterceptor
implements RequestInterceptor {
public void apply(RequestTemplate t) {
String tag = GrayContext.get();
if (tag != null) {
    t.header("X-Gray-Tag", tag);
}
}
}

// RestTemplate
ClientHttpRequestInterceptor  Same logic

// WebClient
ExchangeFilterFunction  Same logic

Why write three times? Because these three outbound abstractions in Spring are completely independent. Real projects most common: Feign added, RestTemplate forgotten — and that forgotten call happens to be query user, so gray order service calls production user service, reads user level computed by old logic. One way to reduce omission: disable unused outbound clients . If team standardizes on Feign, remove RestTemplate Bean from common module, make "wrong outbound" a compile error not runtime silent drift.

4.4 Gateway Side: Weight Predicate and Custom Rewrite

spring:
  cloud:
    gateway:
      routes:
      - id: order-stable_weight
        uri: lb://order-stable
        predicates:
        - Path=/api/order/**
        - Weight=order,90
      - id: order-gray_weight
        uri: lb://order-gray
        predicates:
        - Path=/api/order/**
        - Weight=order,10

Weight predicate rules strict: route id must end with _weight , and same group weights normalized. It solves "split by ratio", cannot solve "same user fixed to gray" — latter needs custom filter: read header or compute user hash, directly rewrite target to lb://order-gray, bypass weight. Practice combines both: whitelist users forced gray via custom filter, rest split by weight. Order: custom filter before RouteToRequestUrlFilter (order < 10000), else URL already built too late — same filter order map as article 04.

4.5 A Hidden Latency: 35 Seconds

Gray instance just registered, max 35s later perceived; scale-down same. So "deploy done, immediately find gray no traffic" 8/10 times this cause, not routing misconfig. Similarly, rollback 35s still requests hit already-down instances. This value tunable ( spring.cloud.loadbalancer.cache.ttl), but don't set to 0 : means every request pulls instance list from registry, dozens of services, thousands QPS will crush registry. Reasonable: keep 5-10s, accept "gray effective has few seconds delay" fact.

Landing: Config, Process, and "How to Prove It Actually Works"

5.1 Gray Group Minimal Config

# application-gray.yml
spring:
  application:
    name: order-service
  cloud:
    nacos:
      discovery:
        metadata:
          version: gray
# Independent namespace or group isolation
server:
  port: 8080

Note version key is convention, not framework built-in. It must exactly match consumer filter read key — case, underscore, spelling, not one char off. Wrong writes no error, only manifests as "filter empty, always fallback all", and if you didn't add fallback metric, this bug can lurk months.

5.2 Five-Step Release Process

① Deploy Gray Group (1-2 replicas)
No traffic, first check startup logs & health
② Whitelist Verification (internal accounts)
Run full business closed-loop, not just one API
③ Small Proportion Ramp (1% -> 5% -> 20%)
Each tier at least cross one traffic mini-peak
④ Observe Four Metrics
Error Rate / P99 / Dependency Errors / Resources
⑤ Full or Rollback
Full = Scale Gray Group + Shrink Production Group

Step ③ "cross a traffic mini-peak" hard requirement. Many issues only appear under concurrency: pool exhaustion, thread queue, cache breakdown, slow SQL 20ms at low peak. Validating at 2am low peak = no validation. Step ⑤ detail often messed: full is NOT upgrading production group , but scaling gray to full, then shrinking production. Version switch becomes just a scale operation, no new deploy — one less deploy, one less error chance.

5.3 Data Isolation: Three Approaches, Cost Differs Greatly

Same DB Same Table
Simplest, rely on code compatibility
Risk: Dirty data mixes, rollback hard to clean

Same DB Different Tables (tag column / shadow table)
Distinguishable, queries must carry condition
Risk: Miss condition -> data cross

Independent Shadow DB
Safest, can discard anytime
Cost: Must sync base data

Selection key not "which safer" but "how to handle gray-produced data on rollback" . If gray writes real orders, rollback those orders remain, old version reading them breaks; if shadow DB, rollback drop DB clean, cost is solving "base data (products, users) how to sync". Practical compromise: Read-only APIs same DB, Write APIs shadow table or shadow DB . Because read risk "see dirty data", clear cache recovers; write risk "produce dirty data", may need manual correction. Treat asymmetric risk differently, more cost-effective than one-size-fits-all.

5.4 How to Prove Traffic Really Went Gray

Easiest step to skip, yet most critical. Because "gray looks normal" two possibilities: new version truly stable, or hardly any traffic went through . Both monitoring curves identical — flat.

① Response Header Echo
X-Served-By: order-v2 / pod-7f9c
② Logs Carry Version
Pattern add %X{appVer}
③ Metrics Split by Tag
http.server.requests{version=v2}
④ Gray Traffic Proportion Dashboard
Target Ratio vs Actual Ratio, Deviation Alert

④ is core: Make "actual gray traffic proportion" a curve, plot with "target proportion" together. Two lines long-term not touching -> routing not effective (or like Incident 2 eaten by connection stickiness); lines touch but gray instance QPS still single digits -> denominator too small, validation no statistical significance. This one curve simultaneously falsifies "not effective" and "not validated" two false safeties.

Six Special Topics: Dyeing, Async, Message, Tasks, Data, Rollback

6.1 Three Full-Chain Dyeing Implementations, Each Has Cost

Pass Header Yourself
Controllable, but must modify all outbound
Suits: Few outbound, team unified

Micrometer Baggage
Auto cross-thread, cross-MQ
Cost: Enters every span, size grows

ThreadLocal + AOP
Only valid in-process
Suits: Monolith or single-hop

Baggage looks easiest but hidden cost: Baggage written into every span and every log's MDC . One chain 30 spans, each carries 3 baggage keys, upload volume rises noticeably. And baggage values default propagate cross-service, equals exposing internal tags to all downstreams — if contains tenant ID sensitive info, extra handling needed. Recommended combo: Header as backbone (explicit, controllable, easy debug), Baggage only as carrier at thread-crossing . I.e. "Value source always header, thread switch borrow baggage or snapshot to transition".

6.2 Async & Thread Pools: Where Dyeing Lost Most Silently

// Wrong: Child thread cannot read context
executor.submit(() -> {
String tag = GrayContext.get(); // null
});

// Correct: Decorator Capture + Restore
public class GrayTaskDecorator
implements TaskDecorator {
public Runnable decorate(Runnable r) {
String tag = GrayContext.get();
return () -> {
    GrayContext.set(tag);
    try { r.run(); }
    finally { GrayContext.clear(); }
};
}
}

Three must-follow points. First, finally clear cannot omit — thread pool threads reused, not clearing carries previous request's tag to next request, manifests as "gray ratio mysteriously high". Second, @Async must explicitly specify executor configured with decorator, default SimpleAsyncTaskExecutor doesn't eat decorator. Third, CompletableFuture.supplyAsync also must pass same executor, no-arg version uses ForkJoinPool, also reads no context. More thorough: use Micrometer's ContextSnapshot.captureAll().wrap(), captures TraceId, baggage, MDC together. This is the mechanism from article 07 — Gray dyeing and tracing share same context infrastructure, don't write two sets .

6.3 MQ: Dyeing Must Land on Message Header, Not Consumer Thread

Producer
headers.put("grayTag", tag);
Consumer
String tag = msg.getHeaders()
    .get("grayTag", String.class);
GrayContext.set(tag);
try { handle(msg); }
finally { GrayContext.clear(); }

Why not put dyeing on consumer thread? Because consumer threads reused, a batch messages may come from different version producers. If tag stays in thread variable uncleared, second message processed with first's tag — and this error only appears when "batch mixes two versions", test env almost impossible to reproduce. Another detail: On message retry, keep original tag? Suggest keep, because retry belongs to same business action, should land same version; otherwise "first consumed by v2, retry by v1", both logics inconsistent causing state corruption.

6.4 Scheduled Tasks: Should Not Dye, Instead Restrict

Scheduled tasks belong to no user request, forcing dyeing yields weird result: Gray version scheduled task runs on production data . It has no header to read, no context to inherit, only "tag" source is config file — better directly control whether it runs.

@Scheduled(cron = "0 0 2 * * ?")
@ConditionalOnProperty(
name = "job.enabled",
havingValue = "true")
public void settle() { ... }

# Gray Group: job.enabled=false
# Production Group: job.enabled=true

If indeed need verify new version batch logic, correct: Run once with shadow data, output to shadow table , not let it run on real data. Batch risk differs from online API: online wrong affects one request, batch wrong affects a batch, often "discovered after run", rollback only via reverse scripts.

6.5 Database Compatibility: Expand-Contract Three Steps

Expand  Add New Column (Nullable/Default)
New and Old Code Both Run
Migrate Dual-Write, Async Backfill Historical Data
Verify Consistency
Contract Confirm No Old Version Then Drop Old Column
Usually Next Iteration, Not Current

Easiest to skip in three steps: Migrate's "verify consistency". After backfill, if not compare row count where new column null/inconsistent, discover data incomplete at Contract step — old column already dropped. Another gray-period special convention: DDL Must Execute BEFORE Gray Group Online, Not After . Because old code doesn't use new column, adding early harmless to old code; reverse if deploy code first then DDL, launch moment "code writes column not yet exist". This order reversed = Incident 3.

6.6 Rollback Harder Than Release, Hard Because Three Things Already Emitted

Code      Rollback 30s
Data      Already Written, Need Reverse Script or Manual Fix
Message   Already Delivered, Cannot Recall
Cache     Already Polluted, Must Clean by Key
Session   User Browser Still Holds Old Token

So "rollback plan" cannot just write "redeploy previous version". Real rollback checklist at least includes: Can old version read new data (added column old code ignores, fine; but new enum value in status, old code parse fails), Who consumes already emitted messages (old version doesn't recognize new event type, throws exception into dead-letter), How to clean cache (version prefix different naturally isolates, same prefix must batch delete by business key). One very practical practice: Before Gray, Do a "Reverse Drill" — Take Gray Group Offline, Observe If System Still Works Normally. This instantly exposes "some function only new version has" hidden dependency, more effective than any doc review.

Eight High-Frequency Pits

① Only Gateway Dyed, Service Calls Lost Tag
Symptom: Gray Metrics Normal, Full Release Explodes
② Each Service Judges Gray Again
Symptom: Same Request Jumps Versions, Hard Reproduce
③ Filter Empty List No Fallback
Symptom: Gray Group Offline -> Whole Site 500
④ Config Class Scanned Global
Symptom: All Services Filtering
⑤ Thread Pool Not Clearing Context
Symptom: Gray Ratio Falsely High, Fluctuates
⑥ Gray Period Executed Destructive DDL
Symptom: Full Release Old Instances Can't Write
⑦ Cache Key Missing Version Prefix
Symptom: Deserialization Fail, Fields Null
⑧ Gray & Production Share Same Table
Symptom: Rollback Dirty Data Uncleanable

These eight ordered by "discovery difficulty": First four logic bugs, traces in logs; Last four data bugs, often hours later surface as "some function data wrong", by then nobody remembers gray caused it. So at gray end must specifically scan "abnormal data", not just watch monitoring curves green.

Troubleshooting Three Axes

Step 1  Check Single Request Which Version
Response Header / Log appVer
Can't Get -> Dyeing Broken at Entrance

Step 2  Check Instance List & Metadata
Nacos Console Confirm Groups
Filter Condition Matches Key

Step 3  Check Proportion Curve
Target Ratio vs Actual Ratio
Touch But Volume Small -> Denominator Issue Not Routing

This order deliberate: First verify "single point correct", then "config correct", finally "statistics correct". Reverse (start with dashboard ratio) falls into "ratio wrong so suspect routing" loop, while actual cause may be that curve stats an API that never went gray. Another often ignored check: Confirm The API You're Checking Is Actually In Gray Scope . Gateway splits by /api/order/**, you load-test /api/item/** — ratio of course 0. Such "problem doesn't exist" troubleshooting wastes most time, because everyone assumes premise correct.

Learning Path & Interview Questions

9.1 Four-Stage Learning Path

Beginner (1 week): Locally run two versions, use Nacos metadata grouping, write simplest supplier filter, two instances verify "tagged request only lands gray group". This step just run once, rest all details.

Advanced (2 weeks): Connect dyeing through Feign, Thread Pool, MQ three points; craft one "filter empty" see if fallback works; Make actual gray traffic proportion dashboard, compare with gateway configured ratio.

Deep (2 weeks): Read three source codes — LoadBalancer supplier composition, Gateway Weight predicate implementation, Feign Interceptor Chain; Clarify "Why Filter Must Be After Cache".

Practice (Ongoing): Complete full five-step release including one active rollback; Do one expand-contract DDL change; Cross-check eight pits against your system, see which hit.

9.2 Twelve Interview Questions (With Summary Answers)

Q1 · Difference Between Gray Release and Blue-Green Deployment?

Answer: Gray routes small real traffic to new version, gradually ramp, goal "verify stability with small risk", switch gradual; Blue-Green maintains two full envs, verify then one-time cut traffic, goal "rollback fast enough", cost double resources. Key difference in risk exposure: Gray "slowly expose", Blue-Green "expose all at once but instantly retractable".

Q2 · Why Only Gateway Gray Is Not Enough?

Answer: Because gray needs "tag follows request". Gateway split only decides first hop which instance, service-to-service Feign calls without propagating tag, second hop falls back to default production instances, forming "new version calls old version" hybrid chain. Such gray validates a non-existent combo, metrics no reference value, and new-old data format diff directly causes deserialization failure.

Q3 · Why Instance Ratio ≠ Traffic Ratio?

Answer: Because chain has pile of "stickiness" factors: HTTP keep-alive & connection pool reuse prevent request reallocation, LB instance list cache (default 35s) delays perception, JVM DNS cache default permanent, session affinity pins same session to instance. Plus traffic itself uneven (big customers, hot keys), actual deviation usually multiples to tens of times. Precise ratio must decide at entrance per request.

Q4 · How to Implement Full-Chain Dyeing?

Answer: Entrance decides once writes into request header (e.g. X-Gray-Tag), thereafter all nodes only read not judge. Propagation relies on four places: HTTP Filter reads, Outbound Client injects (Feign Interceptor / RestTemplate Interceptor / WebClient Filter all three), Thread Pool uses TaskDecorator or ContextSnapshot Capture/Restore, MQ puts in Message Header Consumer Restores. Essentially same context mechanism as TraceId propagation, should reuse not rewrite.

Q5 · Async Thread Loses Dyeing How to Solve?

Answer: Three means: Configure TaskDecorator on Thread Pool, capture context at submit, set before run, finally clear; @Async Must Explicitly Specify Decorated Executor; CompletableFuture.supplyAsync Pass Same Executor, No-Arg Version Uses ForkJoinPool Reads No Context. Most Thorough: ContextSnapshot.captureAll().wrap(), Captures TraceId & MDC Together. Note Clear Cannot Omit, Else Thread Reuse Pollutes Next Request.

Q6 · What DB Compatibility Handling During Gray?

Answer: Three: Schema Only Expand Not Contract (Add Column Nullable/Default, Drop/Change Type Push Next Iteration); DDL Before Gray Online, Because Old Code Doesn't Use New Column Harmless, Reverse Deploy Code First Then DDL Writes Non-Existent Column; Changes Follow Expand-Contract Three Steps, Middle Dual-Write & Backfill Must Verify Consistency, Else Drop Old Column Discover Data Incomplete.

Q7 · What to Watch for Cache During Gray?

Answer: Mainly Serialization Compatibility. Same Key Mixed Two Versions Written Formats, Old Version Reads New Format May Deserialize Fail or Fields Null. Two Solutions: Key With Version Prefix Natural Isolation (Recommended, Rollback Old Data Auto Miss), Or Guarantee Serialization Strict Backward Compatible (Disable FAIL_ON_UNKNOWN_PROPERTIES, New Fields Must Have Defaults). Also Rollback Consider Cache Polluted, Need Batch Delete By Business Key Or Rely Version Prefix Isolation.

Q8 · Where Should LoadBalancer Instance Filter Be Placed?

Answer: Must Be After CachingServiceInstanceListSupplier. Because Cache Must Store Full List, If Filter Before Cache, Cache Only Gray Instances, Production Traffic Reads Cache Gets No Available Instances Direct 500. Placed After, Each Request Does One In-Memory Filter, Dozens Instances Overhead Negligible. Also Custom Config Class Must Not Fall In @ComponentScan Range, Use @LoadBalancerClients Isolate, Else Pollutes Globally.

Q9 · Gray Group Has No Matching Instance How to Handle?

Answer: Fallback To All Instances, Simultaneously Emit Warn Log & Metric. Reason: Gray Is Better Path Not Only Path, Failing Requests To Force Gray Turns Optimization Into Failure. Trigger Scenarios Common: Gray Group Offline, Scaling Not Registered, Some Downstream No Gray Version, Metadata Mistyped. But Fallback Must Be Observable, If Daily Fallback, Gray Strategy Never Effective.

Q10 · How to Prove Gray Traffic Really Hit New Version?

Answer: Four-Layer Verification: Response Header Echo Service Version & Instance (X-Served-By); Log Pattern Carry Version For Per-Request Trace; Metrics Tag By Version For Comparison; Most Important: Plot "Actual Gray Traffic Proportion" And "Target Proportion" On Same Graph. Lines Don't Touch -> Routing Not Effective Or Eaten By Connection Stickiness; Touch But Absolute Value Single Digits -> Denominator Too Small, Validation No Statistical Significance.

Q11 · Why Rollback Harder Than Release?

Answer: Because Release Only Moves Code, Rollback Must Handle Four Things Already Emitted: Written Data (Need Reverse Script Or Manual Fix), Delivered Messages (Cannot Recall, Old Version May Not Recognize New Event Type -> Dead Letter), Polluted Cache (Must Clean By Key), User Browser Still Holds Old Token. So Rollback Plan Must Include "Data What To Do", And Recommend Pre-Gray Reverse Drill: Offline Gray Group, See If System Works, Instantly Exposes Hidden Dependencies.

Q12 · When Not Worth Full-Chain Gray?

Answer: When Release Frequency Low (e.g. Weekly), Service Count Few (3-5), Chain Basically Single-Hop, Full-Chain Dyeing Engineering Effort (Three Outbound Interceptors, Thread Pool Decorator, MQ Header, Metrics Refactor, Dashboard) Far Exceeds Benefit, "Independent Gray Deployment + Gateway Weight" Sufficient. Conversely, Multiple Releases Daily, Dozens Services, Deep Call Chains, Without Full-Chain Dyeing Every Release Is A Gamble. Criterion: "Release Accident Cost × Frequency" > Build Cost.

9.3 One-Page Cheat Sheet

Three Layers
Entrance Split / Instance Group / Full-Chain Dyeing
Only Third Layer Called Gray
Decision Only Once
Entrance Decides, Rest Only Read Not Judge
Multi-Level Decision Needs "Already Decided" Marker
Five Breakpoints
HTTP / In-Process / Outbound / Thread Pool / MQ
Isomorphic With TraceId Propagation
Three Iron Laws
Interface Only Add Not Change
Schema Only Expand Not Contract
Message Format Dual-Read
Fallback
Filter Empty -> Fallback All + Metric
Fallback Daily = Gray Not Effective
Proportion
Instance Ratio != Traffic Ratio
Long Connections / LB Cache 35s / DNS
Data
Read Same DB, Write Shadow
DDL Before Gray Online
Rollback
Code 30s, Data May Need Downtime
Messages Delivered, Cannot Recall

Summary

Gray Release Looks Like Routing Problem, Implementation Is Context Problem, Pitfalls Almost Always In Data. Those Three Incidents: First Tag Didn't Propagate Through, Second Proportion Calculated On Wrong Object, Third Two Versions Can't Coexist — They Correspond To Three Engineering Things: Dyeing, Measurement, Compatibility .

If Only Remember One Sentence: Gray Validates Not "Is New Version Correct" But "Can Old And New Versions Live Together". Code Can Rollback Anytime, Data Cannot. So Design Focus Should Be On "Let Two Versions Coexist At Data Layer", Not "Split Traffic Precisely In Half" — Latter Wrong Just Insufficient Validation, Former Wrong Is Accident.

Next Article Changes Direction, Talk Deployment & Boundaries: "Spring Cloud Cloud Native Deployment & Service Mesh — When To Sink Governance To Infrastructure". Will Clarify What Sidecar Actually Solves, What It Brings, And What Scale Worth Mesh.