Microservices Beyond Theory: Big Tech Production Hard Truths

Core Misconception: Microservices Are Not About Splitting More

Many junior architects believe more services = better architecture . Production reality proves the opposite: blind splitting is the #1 cause of architecture collapse in small-to-medium companies. Over-splitting leads to higher latency, longer call chains, debugging difficulty, distributed transaction proliferation, operational cost explosion, and stability cliff-falls.

Big Tech Architectural Principles

1. Monolith First, Microservices Later — Progressive Evolution

Unwritten big-tech rule: "If a monolith solves it, never blindly microservice."

Suitable for monolith: new/unstable business, small team/fast iteration, low traffic, unsettled domain model.

Suitable for microservices: stable business modules, different iteration rhythms, core paths needing independent scaling/optimization, team splitting for parallel development.

Architecture serves business, not tech showmanship — this separates P6 from P7 thinking.

2. Split by Business Domain (DDD), Not Technical Layers

❌ Wrong: Split by Controller/Service/Dao/Task/Message — ignores business boundaries.

✅ Standard (DDD-driven): Split by business responsibility, boundary, change frequency.

User Center: account, login, permission, membership

Trade Order: place/change/cancel order, state flow

Marketing Activity: coupon, flash sale, group buy, points activity

Inventory Warehouse: pre-deduct, unfreeze, lock, stock alert

Payment Settlement: pay, refund, reconciliation, fund flow

One domain per service → high cohesion, low coupling ; domain-internal changes don't leak outward.

3. Strictly Control Granularity — Reject Over-Splitting

Over-splitting disasters observed in production:

Single order triggers 8–10 remote calls

Any downstream timeout fails entire chain

Retries stack up → retry storms

Distributed transactions everywhere, consistency nightmare

Debugging spans 7–8 log traces

Real big-tech architecture: don't split if avoidable; merge when possible.

Four Production-Grade Hard Modules

① Fault Tolerance: Circuit Breaker, Rate Limiter, Degradation, Bulkhead (Anti-Avalanche Core)

Circuit Breaker (Resilience4j replaces Hystrix)

Purpose: Stop invalid traffic hammering failing downstream.

Production trigger thresholds:

Error rate > 50%

Slow call ratio > 30% (timeout > 1s)

Consecutive failures reach threshold

State flow: Closed → Open → Half-Open Probe → Recover Closed Prevents thousands of futile calls piling threads and dragging upstream down.

Rate Limiting (Big Promo Core Shield)

Token Bucket: Allows burst traffic — used for flash sales, big promos, activity peaks.

Leaky Bucket: Smooths traffic — used for admin, ordinary query APIs, strict even-rate access.

Degradation (Sacrifice Non-Core, Save Core Trade)

Real strategies:

Peak hours: disable points leaderboard, non-essential stats

Product detail page: degrade non-core modules with fallbacks

Exception scenarios: return cached/default data

Core idea: sacrifice secondary functions to keep core transaction chain alive.

Thread Pool Bulkhead Isolation (P7+ Differentiator)

⚠️ Most candidates cannot answer this.

Production: separate thread pools per business/domain:

Order core thread pool

Marketing activity thread pool

Points task thread pool

One business blowing its pool never blocks the core order path. This is the zero-avalanche foundation.

② Service Communication: Gateway + OpenFeign Production Stack

Unified stack across top-tier internet companies: Spring Cloud Gateway (gateway layer) + OpenFeign (inter-service calls) .

Gateway Core Capabilities

Unified entry, routing

Global auth, token validation, IP blacklist

Global rate limiting (IP, API)

Canary release, traffic splitting

Request logging, trace ID propagation

Param sanitization, anti-scraping

Gateway runs on WebFlux async non-blocking — performance crushes legacy Zuul, the only production choice today.

OpenFeign Production Pitfall Avoidance

⚠️ Many companies' dirty data, duplicate orders, duplicate refunds stem from Feign misconfiguration.

Must-configure in production:

Connection timeout & read timeout — strictly separate

Idempotent APIs may retry; non-idempotent APIs must NEVER retry

Retry count & backoff tuned reasonably

Global exception interception & wrapping

All remote calls must log: input, output, trace ID

Ordinary devs just call Feign; senior devs govern Feign risks.

③ Distributed Transactions: The Hardest Bone (Real Business Selection)

Soul question: Cross-service order, inventory deduction, points deduction, payment success — how to guarantee consistency?

Core truth: No perfect distributed transaction — only the optimal fit for your business.

1. Eventual Consistency (90% of Business First Choice)

Scenarios: order, points, activity, notification, statistics

Implementation: Reliable message queue + local message table + state machine compensation

Advantage: High performance, lock-free, high throughput, fits high concurrency

2. Seata Three Modes — Production Differences

AT Mode: Zero code invasion, auto rollback, fits most ordinary business. Drawback: Empty rollback & suspension issues — need production guard logic.

TCC Mode: Manual Try/Confirm/Cancel logic. Fits: Finance payment, refund, reconciliation, fund core paths. Pros: Strongest consistency, zero dirty data.

Saga Mode: Fits ultra-long chains, multi-service long transactions.

Advanced talking point: I use a layered transaction strategy in production: ordinary e-commerce trades use Seata AT for iteration speed; finance/fund/refund/reconciliation core paths forced TCC manual compensation to dodge AT risks; non-core business all use message eventual consistency for high throughput.

④ High Availability Architecture: Big Tech 99.99% Stability Blueprint

P7-level complete answer:

Stateless services: No local data — session, state, cache pushed to Redis/DB → supports instant scale, zero-downtime deploy.

Multi-instance clusters: Core services ≥ 2–3 instances — zero single point, single instance death = zero business impact.

Registry dynamic awareness + health probes: Nacos real-time heartbeat → auto-eject unhealthy nodes → traffic never hits faulty instance.

Multi-region, multi-AZ deployment: Core business cross-region disaster recovery — single region failure auto-switch, business unaware.

Full-chain monitoring + log tracing: SkyWalking end-to-end tracing → pinpoint slow APIs, timeout calls, exception chains → fault localization in seconds.

Canary + rolling deploy: Ban big-bang releases → gray traffic verify, fast rollback on issue .

Senior Java Developer Architectural Mindset

Same microservice talk — 3-year CRUD dev vs 10-year architect gap:

No tech showoff, no over-design: New business never starts microservice; validate market fast, refactor after stability → avoids over-engineering bloat.

Strict call-chain depth control: Ban unlimited nested remote calls; local aggregation over remote loops → cuts network jitter & timeout probability.

Full-chain exception governance: All remote calls unified: timeout control, exception capture, circuit breaker fallback, log instrumentation, monitoring alerting. Exceptions aren't found by humans — systems auto-detect, auto-protect.

Big-promo traffic layered defense: Gateway rate limit → business cache → DB anti-penetration → queue peak-shaving → layered protection absorbs massive traffic spikes .