Service Mesh vs SDK: When Should Governance Move to Infrastructure?

Three Real Incidents Opening the Discussion

Incident 1: Governance gap after polyglot adoption. Core trading services used Java with Sentinel for circuit breaking. Later, data team used Go and algorithm team used Python. During full-chain stress testing, circuit breaker rules covered only 40% of traffic — Go and Python services had neither circuit breaking nor retry, passing pressure directly to the database. Fixing this required writing SDKs for each language and coordinating versions across three teams. This is the problem Service Mesh aims to solve.

Incident 2: P99 latency doubled after Mesh adoption. Test environment latency rose from 8ms to 19ms. Three root causes: two extra proxy hops, new TLS connection per request (no connection reuse), and every request passing through authorization policy checks. Mesh latency is not a fixed 1ms; it depends on connection reuse rate and policy complexity . Without optimizing these, an extra 10ms is normal.

Incident 3: Pods stuck in restart loop after sidecar injection. Kubelet readiness probes targeted Pod IP, which iptables redirected to the not-yet-ready sidecar. Sidecar waited for app readiness; app waited for probe to pass — deadlock. Probes must be excluded from interception — this is Mesh deployment lesson one and the easiest to forget.

Three incidents map to three judgments: Whether to sink governance depends on language stack uniformity, whether latency budget adds up, and whether operational details are mastered. The third is most underestimated — Mesh turns one observable point per call into three, and troubleshooting cost rises for real.

Technical Analysis: Three Generations of Governance and Their Capability Boundaries

2.1 Three Generations: SDK, Sidecar, eBPF

Gen 1  SDK (Spring Cloud)
Governance logic in-process
Strong control, language-bound, upgrade requires code change

Gen 2  Sidecar (Istio / Linkerd)
Governance logic in same-Pod proxy process
Language-agnostic, two extra hops, troubleshooting scattered

Gen 3  eBPF (Cilium etc.)
Governance logic in kernel
One less hop, L7 capabilities still evolving

These three generations are not replacements but three different cost redistributions . SDK puts cost on "write once per language"; Sidecar puts cost on "two extra hops per call, two extra observability points"; eBPF puts cost on "kernel dependency, poor portability". There is no free governance — only which cost you are willing to pay.

An often overlooked fact: SDK has the strongest control of the three . It can read method parameters, catch business exceptions, and make decisions within a transaction boundary. Sidecar only sees HTTP/gRPC layer, so it can do "circuit break if this endpoint returns 500" but cannot do "don't count failures caused by insufficient user balance as circuit-breaker triggers".

2.2 Which Capabilities Should Sink, Which Must Stay in Application

Suitable for sinking:
mTLS, certificate rotation
Routing (by header/weight/version)
Timeout, retry, connection pool
L4 rate limiting, coarse-grained L7 rate limiting
Metrics collection, access logs
Traffic mirroring, fault injection

Must stay in application:
Business circuit breaking (depends on business semantics)
Idempotency, deduplication, transactions
Canary "decision" (can execute, not decide)
Distributed locks, business rate limiting

The sole criterion: Does this decision need business semantics? Whether to retry, how long to timeout — these judge "how important is this call", which only the application knows. "Should two Pods encrypt traffic" or "should this request go to v1 or v2" are pure network-layer problems; infrastructure does them better.

Canary deserves special mention: Mesh can execute stained routing (read header to decide destination) but cannot decide who gets stained. Decision depends on user identity, whitelist, business rules — only available at gateway or application. Pushing decision into Mesh turns into a pile of if-else in YAML — configuration hell, not governance.

2.3 Mesh Cost Accounting, Item by Item

Latency      Same-node ~0.5-1ms
Cross-node 1-3ms
No connection reuse +3-8ms
Complex policies +1-2ms

Memory       50-100MB per sidecar
200 Pods = 10-20GB

CPU          ~0.2-0.5 cores per 10k QPS

Troubleshooting  3 observability points per call
Error codes translated once by proxy layer

The latency column is most misread. Many teams conclude "Mesh added 10ms latency", but connection reuse disabled accounts for the bulk. HTTP/1.1 short connections + mTLS is a disaster combo : each request does two handshakes, one full TLS negotiation = 2-3 RTTs. Enable connection pools and HTTP/2, this drops to near zero.

Memory column is often omitted from capacity planning. 200 Pods × 80MB sidecar = 16GB pure overhead — not counting control plane. If cluster scale is small and service count low, this overhead often outweighs "unified governance" benefits.

2.4 What Scale Justifies Mesh

Services < 10, single language
Not worth it; SDK faster and cheaper

Services 10-30, going polyglot
Can pilot; start with mTLS and observability

Services > 30, or 3+ languages
Worth it; SDK maintenance cost exceeds threshold

Compliance requires (mTLS / audit)
Worth it; hard requirement, not optimization

The core of this table is not the numbers but "SDK marginal cost" . One language = one governance codebase; three languages = three codebases, and every upgrade coordinates three releases. When "onboarding a new service means integrating three SDKs" becomes routine, Mesh economics start to make sense.

Conversely, if you are a single Java stack, ~10 services, few releases per year, Spring Cloud full suite's maturity, troubleshooting convenience, and team familiarity all beat Mesh. Adopting Mesh for "technical advancement" is the most expensive technical debt — its operational complexity will consume 1-2 FTEs long-term.

2.5 Mesh Is Not a Spring Cloud Replacement

Mesh handles:
East-west traffic: service-to-service
Encryption, retry, timeout, routing
Traffic metrics, access logs

Spring Cloud handles:
North-south ingress (Gateway)
Business orchestration, transactions, idempotency
Config, messaging, task scheduling

These are layered, not mutually exclusive. Real production: Gateway (north-south) stays Spring Cloud Gateway because auth, rate limiting, protocol translation need business context; service-to-service calls (east-west) go to Mesh because that layer is most standardized, least needing business semantics.

Mixing without clear boundaries causes blame-shifting during incidents. Clear boundary: "Cross-process network behavior belongs to Mesh; in-process business behavior belongs to application" . Once this line is drawn, ownership of retry, timeout, circuit break falls naturally.

2.6 Three Deployment Topologies, Different Pain Points

Single cluster
Control plane + data plane same cluster
Trust domain consistent, simplest

Multi-cluster
Each cluster own control plane
Or shared control plane + remote data planes
Pain point: trust domains must connect

Hybrid (incl. VMs)
VM services have no sidecar
Need VM-side proxy
Or gateway protocol bridging

Cross-cluster blocker is rarely network connectivity — it's identity . mTLS certificates carry SPIFFE identities containing trust domain (default = cluster name). Two clusters issue separate certs; mutual verification fails due to trust domain mismatch — network up, ports open, handshake fails.

Fix requires trust domain federation: exchange root CAs, explicitly declare trust. This is easily missed during migration because single-cluster tests never reveal it. If your target is multi-cluster, this must be designed before first cluster goes live , otherwise adding clusters equals redoing everything.

Core Flow: How a Request Travels Inside Mesh

3.1 Full Path: App to App Through Four Gates

① App initiates request
   Target: order-svc:8080
② iptables intercept (OUTBOUND)
   Redirect to 127.0.0.1:15001
③ Envoy outbound
   Select cluster (EDS provides endpoints)
   Do mTLS, retry, timeout
④ Peer Envoy inbound (15006)
   Verify mTLS, auth policies
   L7 metrics, access logs
⑤ Forward to app (localhost:8080)

Note steps ② and ⑤: Application is completely unaware of proxy existence — it thinks it connects directly to order-svc. This transparency is Mesh's value and also the root of troubleshooting difficulty — application logs show target address, but actual connection goes elsewhere.

Another detail: step ④ inbound also does policy check. Policies are bidirectional — Service A allowed to call B doesn't mean B allows being called by A; both sides must configure. This is a Mesh advantage over SDK: SDK era "who can call me" was inexpressible except via network isolation.

3.2 Traffic Interception: Why Probes Get Hijacked

# Ports to exclude
15020  merged metrics
15021  health check
15090  Envoy admin

# Probe uses Pod IP, not localhost
# Source address not in 127.0.0.1 range
# -> hits inbound rule
# -> forwarded to not-yet-ready sidecar

This is Incident 3's mechanism. Kubelet probes from node targeting Pod IP:8080; iptables classifies as "inbound traffic", redirects to 15006 for Envoy. Envoy readiness depends on app readiness (it has readiness port); app readiness depends on probe passing — deadlock.

Three solutions, first recommended: 1. Annotate to exclude probe ports from interception (annotation names differ; Istio uses traffic.sidecar.istio.io/excludeInboundPorts); 2. Start sidecar before app ( holdApplicationUntilProxyStarts); 3. Change probe to exec mode (run inside container, hits localhost). Third is easiest but masks the problem — probe no longer tests network reachability.

3.3 Control Plane: What xDS Actually Pushes

LDS  Listener        Which ports to listen, which filter chain
RDS  Route           Path match -> which cluster
CDS  Cluster         Service definition (timeout/retry/circuit break)
EDS  Endpoint        Instance list (IP:port)
SDS  Secret          Certificates and keys

Among five resources, EDS changes most frequently — instance up/down, scaling, health status changes all trigger it. This is where Mesh meets service registry: if discovery is still Nacos, EDS data must sync from Nacos; if services already in K8s, reading Service Endpoints directly is simpler.

xDS is eventually consistent, not strongly consistent — must accept this. Control plane push has latency, Envoy config apply has latency, old and new configs may coexist briefly. So "just scaled up then stress test" seeing uneven traffic is often not LB algorithm issue but EDS not fully propagated — same class as LoadBalancer caching 35 seconds discussed in previous article.

3.4 A Working Routing Configuration

apiVersion: networking.istio.io/v1
kind: VirtualService
spec:
hosts: [order-svc]
http:
- match:
  - headers:
    x-gray-tag:
    exact: v2
  route:
  - destination:
    host: order-svc
    subset: v2
- route:
  - destination:
    host: order-svc
    subset: stable
    weight: 90
  - destination:
    host: order-svc
    subset: v2
    weight: 10
timeout: 2s
retries:
  attempts: 2
  perTryTimeout: 800ms

This config maps 1:1 to previous article's canary scheme: header match = "route by label", weight = "by proportion". Only difference: rules moved from Java code to YAML . So previous article's three iron laws (stain propagation, proportion by traffic, bidirectional compatibility) apply unchanged — governance logic changed location, engineering constraints didn't.

Note final timeout and perTryTimeout: Total timeout 2s, single try 800ms, two retries . Calculation must be 800×2 + first 800 ≤ 2s, otherwise third attempt has no time to complete. If this budget isn't clean, retry is "configured but useless".

3.5 What Happens When Control Plane Dies

Existing calls
  Continue, using last config snapshot
Config changes
  Stop propagating, frozen
New instances coming online
  Cannot register, scaling ineffective
Scaling / route changes
  All paused

This design is called control-plane/data-plane failure domain separation — prerequisite for production Mesh: control plane crash doesn't affect existing traffic because each sidecar holds full local config snapshot. But "no impact" has boundaries — you lose change capability . During outage, want to adjust route, drain instance, scale — all impossible.

So control plane HA cannot be compromised: 3 replicas minimum, cross-AZ, etcd deployed separately with backups. More importantly, drill "control plane fully down" — verify existing calls truly unaffected, verify config auto-catches up on recovery. Undrilled DR = no DR.

Core Mechanisms: Envoy's Three-Layer Filters and Spring Cloud Adaptation

4.1 Envoy's Three-Layer Structure

Listener (port)
-> Network Filter Chain
   TLS termination / mTLS verification
-> HTTP Connection Manager
-> HTTP Filter Chain
   Auth / Rate limit / Routing / Tracing
-> Router (select cluster)
-> Cluster (upstream connection pool)

This structure is isomorphic to Spring Cloud Gateway's filter chain — both "layered + chained + order-sensitive". Difference: Gateway chain runs in JVM, reads Spring Beans; Envoy chain runs in C++ process, reads only config. Capability boundary determined by "can it get context" — this rule runs through the entire article.

TLS at outermost layer (network filter) means mTLS verification happens before any L7 logic . Good: unauthenticated connections never reach HTTP layer, smaller attack surface. But creates troubleshooting issue — cert problems appear as connection resets in Envoy logs; application sees only "connection closed by peer".

4.2 Why Some Spring Cloud Capabilities Must Be Actively Disabled

LoadBalancer
  Mesh EDS already does it
  App just connects to Service name

Resilience4j Retry
  Multiplies with Envoy retry
  3 x 3 = 9 times

Sentinel Rate Limiting
  Coarse-grained can sink
  Business-grained keep in app

Second item is the classic accident. App configures 3 retries, Mesh configures 3 retries — one failure amplifies to 9 real requests , downstream sees 9x designed traffic. This hides in stress tests — you push 1000 QPS, downstream sees 9000.

So migration principle one: Same capability must have exactly one effective location . Approach: disable app-side retry first (keep circuit breaker for business semantics), observe one week confirming Mesh retry policy suffices, then decide whether to move circuit breaker down. Move one capability at a time, observe one week per move — this discipline matters more than any architecture diagram.

4.3 Three Service Discovery Integration Modes

① K8s Service direct
   EDS reads Endpoints
   Simplest, requires all services in K8s

② Registry sync
   Nacos -> Control plane -> EDS
   Compatible with VMs and hybrid

③ Dual registration coexistence
   SDK and Mesh each hold a copy
   Migration use only, long-term burden

Migration often uses third mode, cost: "two instance lists may diverge". Symptom subtle: some calls go Mesh (get new instances), some go SDK (get old list) — "occasionally can't connect to newly scaled instance". Dual registration must be transitional with explicit exit deadline , otherwise becomes permanent architecture nobody dares touch.

4.4 A Must-Hit Pitfall: Readiness Out of Sync

Sidecar ready and app ready are separate, but traffic entry only watches sidecar. So window appears: "sidecar ready, app not up yet" — few hundred milliseconds of total failures. Istio provides holdApplicationUntilProxyStarts to delay app start until sidecar ready, but only solves half — reverse "app ready but sidecar config not pushed" still exists.

Practice: widen startupProbe failure threshold to give container time to cross window; simultaneously app side must not assume first call succeeds , especially startup logic that pulls config, connects registry. This "startup fragility" is a new failure class introduced by Mesh, absent in SDK era.

Landing: Three Migration Paths and Supporting Changes

5.1 Choosing Among Three Paths

Big bang
  Inject all namespaces at once
  Fast, but risk concentrated, rollback requires removing sidecars

Namespace-by-namespace canary
  Non-core business first, then core
  Recommended, clean boundaries

Traffic-proportion canary
  Same service, some instances with sidecar
  Smoothest, but config alignment hardest

Recommend second. Reason not "lower risk" but "namespace is a naturally rollback-able boundary" — on issue, remove annotation, restart, back to SDK mode, impact scope enumerable. Third looks smoother but actually hardest: same service, some instances with sidecar, some without — traffic randomly jumps between "proxied" and "not proxied", metrics, timeouts, retries all misalign.

5.2 Metrics Must Reconcile

Envoy side
  istio_requests_total
  by source/destination/response_code

App side
  http_server_requests
  by uri/method/status

Critical: both sides must join
  via destination_service + path

Post-Mesh most common confusion: "two monitoring panels show different numbers". Cause usually different statistical口径: Envoy counts connection-layer requests (including rate-limited, policy-blocked), app counts requests entering business code. Delta is "volume blocked by infrastructure" — this delta is valuable but must be communicated upfront, otherwise every postmortem wastes half hour aligning numbers.

Practical tactic: Reconcile Envoy response_code with app status . Normally they should match; mismatches pinpoint "proxy layer intervention" — rate limit, auth failure, upstream connection failure. This reconciliation line locates 80% of "app down or network down" disputes.

5.3 Troubleshooting Methodology Must Evolve

Old: App logs + distributed tracing
New: Three-way alignment
  ① Caller Envoy outbound logs
     Request sent? Where to?
  ② Callee Envoy inbound logs
     Received? Why rejected?
  ③ Both app logs
     Business processing result

Key to three-way alignment: Same request ID traceable in all three places . Requires Mesh access logs carry tracing header, and app log pattern prints same ID — exactly the propagation mechanism from Article 07, whose importance rises in Mesh era because one more layer needs correlation.

Most common misjudgment: treating "Envoy returns 503" as "service down". 503 in Mesh has at least four sources: no healthy upstream endpoints (EDS empty), connection pool full, rate limited, circuit broken. Must check response_flags (UF/UC/UO/NR etc.) to differentiate; staring at status code keeps you searching in wrong direction.

Six Special Topics: mTLS, Retry Multiplication, Timeout Budget, Rate Limit Ownership, eBPF, When Not to Mesh

6.1 mTLS Three Modes and Migration Sequence

DISABLE      Plaintext, nothing done
PERMISSIVE   Accept both plaintext and mTLS
STRICT       Accept only mTLS

Migration: DISABLE -> PERMISSIVE
           -> Observe period
           -> STRICT

PERMISSIVE is critical intermediate state — lets "sidecar-injected services" and "not-yet-injected" communicate. But its existence is risk — any client can bypass encryption via plaintext , so it's transitional, not stable, must have explicit STRICT cutover deadline.

Before switching to STRICT, do one thing: Confirm all callers have sidecars injected . Miss one (e.g., batch job using host network, or legacy cross-cluster service) — cutover moment it all fails. Check method: watch plaintext request metric during PERMISSIVE; if non-zero, don't switch.

6.2 Retry Multiplication: Classic Mesh Landing Accident

App      maxAttempts=3
Envoy    attempts=3
Actual   3 x 3 = 9 times
If both layers retry again x 3 = 27 times

Detection: Watch downstream QPS
Stress 1000, downstream 9000 -> Caught

Why missed? Two configs live in completely different places: app in application.yml, Mesh in VirtualService, maintained by different people. No tool tells you "you configured two layers of retry" — only downstream QPS curve does.

Fix isn't "no retries anywhere" but explicitly retry in exactly one layer, and that layer must sense downstream state . Usually recommend Mesh layer: it distinguishes "connection failed" (should retry) from "business 500" (should not), while app-side retry often doesn't distinguish, amplifying idempotency issues too.

6.3 Timeout Budget: Who Is Final Arbiter

Client Envoy   2s (total budget)
Per try        800ms
App side       1.5s (must be < 2s)
Downstream RPC 800ms
Downstream DB  300ms

Principle: Outer timeout must exceed inner , otherwise outer breaks first while inner still executes — produces "ghost requests": client already timed out, server still running, finishes DB writes, sends messages, creates real side effects. These appear in logs as "timed out but data changed" — hardest to debug.

Correct config order is inside-out: Set DB timeout first, then downstream RPC, then app total, finally Envoy total . Reverse (arbitrary gateway 5s) leaves inner layers without budget concept; any layer slowing down pushes to outer. Same timeout pyramid as Article 7, just with one extra proxy layer.

6.4 Rate Limiting: Mesh or App?

Mesh suitable for:
  Coarse-grained by IP / service / path
  Connection count limits
  Global quota (with external service)

App suitable for:
  Fine-grained by user / tenant / parameter
  Business quota (e.g., daily order limit)
  Decisions needing business context

One criterion: Can this layer obtain the inputs needed for the decision? Mesh only sees headers and path, so it can do "this endpoint max 1000/sec" but cannot do "this user max 10/min" — latter needs token parsing or user table lookup.

When both layers configured, watch threshold direction : Mesh threshold should be larger than app's, so rejection happens in app (more precise). Reverse causes all requests blocked at Mesh, app's fine-grained rules never trigger — same sequencing principle as Gateway rate limiting in Article 04.

6.5 eBPF: What It Solves, What It Doesn't

Solves:
  Bypass iptables, one less hop
  Kernel-space forwarding, lower latency
  Observability granularity to syscall

Doesn't solve:
  L7 policy capabilities still evolving
  Kernel version dependency (4.19+)
  Envoy remains L7 workhorse

eBPF value mainly in data plane efficiency : move forwarding from user-space proxy to kernel, save one context switch. But "governance" core isn't forwarding efficiency — it's policy expressiveness: retry semantics, L7 routing, request-level auth — Envoy still more mature. Real architectures often "eBPF for network + Envoy for L7" combo, not either/or.

For most teams, this layer isn't decision time yet. Master SDK or Sidecar first; that matters far more than chasing new tech — because regardless of generation, when things break you answer same questions: where did this request go, why rejected, whose timeout?

6.6 When Not to Mesh

These signals mean don't adopt:
  Services < 10, single language
  No one can full-time maintain control plane
  Troubleshooting still relies on "restart and see"
  Release frequency < once per week
  Team hasn't unified logging and tracing

Last two are most overlooked but are prerequisites. Mesh splits one call into three observability points; if you haven't solved "can app logs be correlated by traceId", Mesh only adds chaos — you get three uncorrelated log streams, not better observability .

In other words: Mesh is "governance capability amplifier", not "governance capability source". Already-clear architecture becomes clearer; already-messy becomes messier, just moved from code to YAML.

6.7 An Executable Rollout Sequence

① Inject only, enable zero capabilities
   Measure latency and memory delta
② Enable observability (metrics + logs)
   Build three-way alignment capability
③ Enable mTLS: PERMISSIVE
   Observe, then cut to STRICT
④ Disable app-side retry, keep only Mesh
   Watch downstream QPS drop
⑤ Finally move routing and rate limiting
   One capability at a time, observe one week

Sequence cannot be reordered, especially step ①. It looks lowest value — nothing enabled, just extra process — but it gives clean latency and resource baseline . Every subsequent capability added, you can say "this 1ms belongs to X". Skip this, three months later you face an un-attributable latency curve.

Step ④ often skipped because "looks like no change": disabling app retry just removes retries. But if both layers retry, you never measure Mesh retry's true effect — with two layers stacked, all metrics are blended, impossible to evaluate either layer alone .

Eight High-Frequency Pitfalls

① Probe hijacked, Pod won't start
   Exclude probe ports or delay app start
② Two-layer retry multiplies, downstream hit 9x
   Same capability only in one place
③ Outer timeout shorter than inner
   Ghost requests: timeout but data changed
④ PERMISSIVE treated as stable state
   Plaintext bypasses encryption, must go STRICT
⑤ Dual registration long-term coexistence
   Two instance lists diverge, occasional connect failures
⑥ Sidecar ready before app ready
   Startup window requests all fail
⑦ Only watch 503, ignore response_flags
   Four causes, can't tell which
⑧ Metric口径 not aligned
   Two panels differ, postmortem argues half hour

These eight group into three categories: Startup (①⑥), Config (②③④⑤), Runtime (⑦⑧) . First two explode during migration; third persists long after stable. So first month post-launch focus on startup issues; from month three, establish new troubleshooting methodology.

Troubleshooting Three-Step Method

Step 1  Segment
  Caller Envoy outbound: request sent? where?
  Not sent -> App didn't call
  Sent -> Check callee inbound

Step 2  Check response_flags
  UF Upstream connection failed
  UC Upstream connection closed
  UO Overflow (circuit break / rate limit)
  NR No route

Step 3  Align both-side metrics
  Envoy request count vs App request count
  Delta = Volume blocked by infrastructure

Core idea: Binary search . One call passes five points; first decide if problem in first half or second half, then binary search further. Orders of magnitude faster than "pull logs from three places and stare".

One self-check item: First confirm this service actually has sidecar injected . Migration often sees "thought it went Mesh, actually not injected" — all Mesh configs ineffective for it, yet you hunt in VirtualService. Check Pod container count — two containers (app + proxy) means injected.

Learning Path and Interview Questions

9.1 Four-Stage Learning Path

Beginner (1 week): Install Istio on local Kind/Minikube, inject two services, run "header routes to different version". Focus not on memorizing commands but manually inspect iptables rules once to understand how traffic gets hijacked.

Intermediate (2 weeks): Take mTLS from DISABLE to STRICT via PERMISSIVE, observe plaintext request metrics; configure timeout and retry, deliberately slow downstream, verify if "ghost requests" appear; compare Envoy vs app metric differences.

Advanced (2 weeks): Read xDS five resource definitions, understand EDS push timing; examine Envoy config dump, understand listener -> filter chain -> cluster actual structure; memorize common response_flags values.

Practice (ongoing): Do real namespace-by-namespace migration, disable app retry first keeping only Mesh layer, observe one week; build three-way alignment troubleshooting handbook; finally write "when to cut back to SDK" judgment criteria.

9.2 Twelve Interview Questions (with Summary Answers)

Q1 · What problem does Service Mesh solve? Core: solves "governance logic coupled with business code and language-bound". SDK mode requires each language to implement circuit break/retry; upgrades need code changes and coordinated releases across all teams. Mesh moves this to same-Pod sidecar proxy, transparent to app, language-agnostic, upgrades only touch infrastructure. Trades unified transparency for latency, resources, and operational complexity.

Q2 · Mesh and Spring Cloud relationship? Layered, not replacement. Mesh handles east-west (service-to-service) network behavior: mTLS, retry, timeout, routing, traffic metrics. Spring Cloud handles north-south ingress and business capabilities: Gateway auth/rate-limit, config, messaging, transactions, idempotency, task scheduling. Clean boundary: "Cross-process network behavior -> Mesh; In-process business behavior -> App".

Q3 · Which capabilities sink to Mesh, which stay in App? Criterion: "Does this decision need business semantics?" Sink: mTLS/cert rotation, routing, timeout/retry/connection pool, coarse rate limiting, metrics/access logs, traffic mirroring. Keep in App: Business circuit break (needs business semantics), idempotency/deduplication/transactions, canary decision (Mesh executes, cannot decide), distributed locks, fine-grained user/tenant rate limiting.

Q4 · Why Pod won't start after sidecar injection? Kubelet readiness probe targets Pod IP; iptables classifies as inbound, redirects to sidecar; sidecar readiness depends on app readiness; app readiness depends on probe passing — deadlock. Fix: exclude probe ports from interception, or configure sidecar start before app (holdApplicationUntilProxyStarts), or use exec probe (hits localhost). First recommended because others mask network reachability validation.

Q5 · Where does Mesh latency overhead come from? Three parts: Two extra proxy hops (same-node ~0.5-1ms, cross-node 1-3ms); TLS handshake (no connection reuse = 2-3 RTTs per request, the big chunk); Policy checks (more auth/rate-limit rules = slower). Optimization order: enable connection pools + HTTP/2 reuse first, then simplify policies, finally consider data plane swap. Many "Mesh too slow" conclusions are actually short-connection caused.

Q6 · What are xDS five resources? LDS defines Listener (which ports, which filter chain); RDS defines Route (path match -> cluster); CDS defines Cluster (upstream service with timeout/retry/circuit-break); EDS defines Endpoint (instance list, changes most frequently); SDS defines Secret (certs/keys). EDS directly ties to service discovery — Mesh/registry intersection.

Q7 · Why retry multiplication? How to avoid? App (Resilience4j) and Mesh (VirtualService) each configured retry; two layers chain, 3×3=9. Configs in two places, maintained by different people, no tool warns duplication. Symptom: downstream QPS multiples of stress value. Fix: Same capability exactly one effective layer. Recommend keep Mesh (distinguishes connection failure vs business error), disable app retry, observe downstream QPS to confirm.

Q8 · How to configure timeout under Mesh? Inside-out increasing: DB timeout < Downstream RPC timeout < App total timeout < Envoy total timeout. Outer must exceed inner, else outer breaks first while inner still runs -> "ghost requests": client timed out, server still writes DB, sends messages, real side effects. Retry config must also budget: per-try timeout × attempts ≤ total budget. Same timeout pyramid as Article 7, just one extra proxy layer.

Q9 · Mesh metrics don't match — what to do? First clarify口径: Envoy counts connection-layer requests (includes rate-limited, policy-blocked); App counts requests entering business code. Delta = "volume blocked by infrastructure". Join via destination_service + path; reconcile Envoy response_code with App status. Mismatches pinpoint proxy intervention points — rate limit, auth fail, upstream connection fail. This reconciliation line resolves 80% of "app down or network down" disputes.

Q10 · Why must mTLS migration go through PERMISSIVE? Migration period has two service types: sidecar-injected (can present cert) and not-injected (plaintext only). PERMISSIVE accepts both, ensures connectivity. But it's transitional, not stable — plaintext bypasses encryption, so must cut to STRICT. Before switch, confirm PERMISSIVE plaintext metric zero; else everything breaks.

Q11 · Will eBPF replace Envoy? Not short-term. eBPF advantage: data plane efficiency (bypass iptables, kernel forwarding, lower latency). But governance core is policy expressiveness — L7 routing, retry semantics, request-level auth — Envoy still more mature. Real architectures: "eBPF for network + Envoy for L7" combo, not either/or.

Q12 · When not to Mesh? Services < 10 single language (SDK faster/cheaper); No dedicated control plane maintainer; Troubleshooting still "restart and see"; Release frequency < weekly; Logging/tracing not unified. Last two are prerequisites — Mesh splits one call into three observability points; without traceId correlation, only more chaos. Conclusion: Mesh amplifies governance capability, doesn't create it.

9.3 One-Page Cheat Sheet

Three Generations
SDK: Strong control, language-bound
Sidecar: Transparent, two extra hops
eBPF: One less hop, L7 maturing

Sink Criterion
Decision need business semantics?
Yes -> Keep App, No -> Sink

Cost
Per hop 0.5-3ms (connection reuse variable)
Per sidecar 50-100MB
Observability points 1 -> 3

Scale
<10 services single lang: Not worth
>30 services or multi-lang: Worth
Compliance needs mTLS: Hard requirement

Path
Namespace canary, naturally rollback-able
One capability at a time
Same capability only one effective location

Timeout
DB < RPC < App < Envoy
Outer breaks first = Ghost request

Retry
Two layers multiply 3x3=9
Watch downstream QPS to know

Troubleshoot
Binary search segments -> response_flags
-> Two-side metrics reconciliation

Summary and Series Conclusion

Mesh's value proposition was never "faster" or "stronger", but "turn N copies of code into one config" . It trades latency, resources, and troubleshooting complexity for language-agnostic unified governance. Whether this trade pays off depends on your service count, language variety, and team's ability to digest those three observability points.

If only one sentence remembered: Governance capability comes from architecture, not tools . Mesh can move your existing retry, timeout, routing rules to sidecar; it cannot move the judgment of "how exactly do I want to govern". A team without clear timeout budgets, after adopting Mesh, just moves misconfigured timeouts from yml to YAML.

Twelve-article retrospective: 01 System Overview (versions & landscape) -> 02 Nacos (Registry & Config) -> 03 OpenFeign (Calls & Timeouts) -> 04 Gateway (Ingress & Filters) -> 05 Sentinel (Flow Control & Circuit Break) -> 06 Config Canary (Change Safety) -> 07 Distributed Tracing (Context Propagation) -> 08 Resilience4j (Fault Tolerance Combos) -> 09 Gateway Auth (Identity & Trust) -> 10 Stream (Messaging Abstraction) -> 11 Canary Release (Stain Propagation) -> 12 Mesh (Responsibility Boundaries). Twelve articles actually discuss one thing: When a request traverses microservices, who owns its identity, context, budget, and boundaries.

Series ends here. To reread, suggested order: 01 -> 07 -> 11 -> 12: First get landscape, then understand context propagation, then understand staining, finally judge who should do what. Remaining eight articles expand these four; revisit when hitting specific problems. Architecture difficulty never lies in knowing how to use components, but in knowing where each component's responsibility ends.