A Decade of E‑Commerce Ops: How to Prevent System Outages and Ensure High Availability

Having worked on e‑commerce platforms for years, the author shares practical ways to keep systems stable and avoid costly downtime.

Why systems crash (common failure modes)

Traffic spikes (e.g., flash‑sale bursts) that overwhelm servers.

Dependency poisoning where a single slow API drags the whole system down.

Database overload causing CPU to hit 100% and orders to stall.

Human error such as accidental rm -rf deletions.

Four‑layer defense architecture (like house construction)

Layer 1: Entry protection

Load balancer distributes requests across multiple machines.

Rate limiting (e.g., allow 1,000 requests per second, excess queued).

Example: during a flash sale, show a “queueing” message instead of crashing.

Layer 2: Service autonomy

Deploy each service independently so one failure does not affect others.

Key technique: service degradation.

public ProductDetail getDetail(Long productId) {
    // Normal: fetch inventory, reviews, recommendations
    // During big promotion: only fetch basic info
    if (isBigPromotion()) {
        return basicInfoOnly(productId); // degradation!
    }
}

Layer 3: Data protection

Cache strategy:

Store hot product data in Redis to relieve DB pressure.

Beware of cache‑penetration when querying non‑existent keys.

Database backup:

Master‑slave replication (writes to master, reads from slaves).

Automatic daily backups retained for 30 days.

Layer 4: Fallback mechanisms

Static pages: show a maintenance page if the system is down.

Financial operations must go through a reconciliation system to ensure no money is lost.

Monitoring and alerting (system cameras)

Essential metrics to monitor:

Server CPU and memory usage (alert >80%).

API response time (alert >1 s).

Error logs (instant SMS on exceptions).

Business KPIs: order success rate, payment success rate.

Simple monitoring code example:

long start = System.currentTimeMillis();
try {
    // business logic
} finally {
    long cost = System.currentTimeMillis() - start;
    if (cost > 1000) { // over 1 second
        log.warn("Interface slow warning: cost={}ms", cost);
    }
}

Emergency handbook (fire drill)

Contact list (Ops, DBA, owners) with phone numbers.

One‑click degradation switch for big promotions.

Rollback scripts (e.g., revert a bad release within 5 minutes).

Communication templates for customer service.

Real‑world scenario responses

Scenario 1: Sudden traffic during a promotion

Pre‑scale servers before the event.

Gradual ramp‑up: start with 10 % of users, then increase.

Cart purchase limits to block malicious orders.

Scenario 2: Database suddenly slows

Failover: promote a replica to master immediately.

Temporarily suspend non‑critical features (e.g., product reviews).

Identify the culprit, usually a slow SQL, and optimize it.

Scenario 3: Third‑party payment service outage

Guide users with a “payment system busy” notice.

Record orders locally and notify users once restored.

Compensation: offer a small coupon as an apology.

Final hard truths

No system can be 100 % never‑down; aim for rapid recovery.

Keep it simple: three 9’s (99.9 % uptime) is sufficient; chasing four 9’s adds unnecessary complexity.

Conduct quarterly disaster‑recovery drills.

Documentation is critical; staff turnover should not erase knowledge.

Technical key points

Degradation strategies should be business‑driven; non‑core functions can be cut.

Set reasonable thresholds for monitoring to avoid false alarms.

Test database backups regularly to ensure they are usable.

Third‑party dependencies must have timeout settings; otherwise they can cause indefinite hangs.