How JD Cloud’s Log Service Powered the Record‑Breaking 11.11 Sale

Overview

During JD.com’s 11.11 Global Shopping Festival, the JD Cloud Log Service handled petabyte‑scale log data, supporting load balancers, DNS, CDN and providing real‑time monitoring for the entire system.

Functional Requirements

On‑site log viewing

Log monitoring

Real‑time analysis

Log consumption

Cost Considerations

Log data generated daily reaches petabytes, making storage cost a critical factor. Object storage with low‑cost, elastic scaling and S3 compatibility was chosen, and both storage media and content are compressed to reduce expenses.

Stability Measures

The service must remain highly available during the peak event. Architecture separates storage and compute, uses multi‑AZ/region redundancy, lightweight real‑time compute, big‑data engines with HA and checkpointing, and implements circuit breaking, rate limiting, auto‑scaling, and comprehensive monitoring dashboards.

Architecture Details

Three‑layer design:

Collection layer : agents collect logs from hosts, containers, etc.

Storage layer : leverages database and object storage for durable, highly available storage.

Compute layer : filters, transforms, aggregates logs for downstream services.

Business layer : provides search, analysis, monitoring, and consumption features.

Storage uses object storage with features such as elastic scaling, low cost, high availability, security, and S3 compatibility. Logs are indexed and bucketed, both compressed.

Compute loads required buckets, decompresses blocks, performs indexing and aggregation using priority queues and worker pools.

Monitoring and Pre‑plan

Dashboards (screen‑monitoring and ops versions) display global health, key metrics, and alerts. Monitoring covers overall SLA, golden metrics (PV, latency, error codes), and synthetic checks. Pre‑defined high‑availability, circuit‑break, rate‑limit, scaling, and fallback plans are prepared and rehearsed.

Conclusion

The JD Cloud Log Service’s robust architecture, cost‑effective storage, and comprehensive stability mechanisms ensured the 11.11 event ran smoothly, delivering lower costs and higher efficiency.