This article traces the history of Linux's source‑code management, explains why Linus Torvalds created Git as a distributed version‑control system, compares it with centralized systems like CVS and SVN, and highlights Git’s key advantages and ecosystem.
This article explains why Kafka can be used for permanent data retention, outlines practical use cases such as event logging, cache rebuilding, stream recomputation, and database change capture, and clarifies why Kafka is a stream‑processing platform rather than a traditional message queue or database.
PhxQueue, an open‑source, Paxos‑based distributed queue from WeChat, delivers at‑least‑once delivery, synchronous disk flushing, strict ordering, multi‑subscription, and high availability, outperforming Kafka in reliability and latency while maintaining comparable throughput, as demonstrated through detailed design, performance, and failover analyses.
This article recounts Xiaomi's rapid development of a high‑concurrency flash‑sale platform for the 2014 "Mi Fan Festival", detailing the design choices, trade‑offs, and performance optimizations that enabled millions of users to purchase phones simultaneously.
The article presents the architecture, component design, optimizations, and reliability guarantees of Bilibili's Lancer log collection system, a Flume‑based distributed pipeline that handles both real‑time and offline data streams for billions of events daily.
This article outlines the progressive stages of system architecture—from a single‑server LAMP setup through application‑data separation, caching, clustering, read/write splitting, CDN, distributed storage, NoSQL, business decomposition, and finally distributed services—while detailing essential technologies such as message queues, service frameworks, service buses, communication patterns, and governance mechanisms like Dubbo and OSB.
This article explains how a web system can evolve from handling 100,000 daily visits to over 100 million by progressively implementing multi‑level caching, various load‑balancing techniques—including HTTP redirects, reverse‑proxy, IP‑level, DNS, and GSLB—and optimizing MySQL through indexing, connection pooling, sharding, replication, and integrating memory caches such as Redis to ensure high performance and reliability.
This article explores the core concepts, design patterns, and practical techniques—including caching, distributed coordination, front‑end scaling, and back‑end queuing—to build robust high‑concurrency flash‑sale systems while preventing oversell and performance bottlenecks.
In this talk, chief architect Li Yanpeng shares his career background, outlines the goals and design principles of distributed service architecture—including high availability, performance, scalability, extensibility, security, and consistency—and offers a methodology for cultivating both technical and personal inner skills for engineers.
This article presents a comprehensive design for a high‑traffic insurance O2O platform, detailing functional requirements, system analysis, storage and caching strategies, logical and service architectures, distributed transaction handling, and the chosen development stack, emphasizing simplicity, scalability, and high concurrency.
This article dissects the technical challenges of building a high‑concurrency flash‑sale (seckill) system—covering business analysis, traffic isolation, static page caching, CDN bandwidth, dynamic order URLs, request throttling, database sharding, optimistic locking, and anti‑cheat mechanisms—while presenting concrete architectural principles and code examples.
The article provides a comprehensive overview of Ceph, covering its object storage concepts, architecture, core technologies, performance testing results versus Swift, and concludes with a summary of findings and future outlook for this distributed storage solution.
This article reviews the evolution of Meituan‑Dianping's MySQL high‑availability architecture over recent years, detailing the transition from the MMM replication manager to MHA, the integration of Zebra and Proxy middleware, and future design considerations such as distributed agents, semi‑synchronous replication, and MySQL Group Replication.
This article traces Meituan‑Dianping's MySQL high‑availability journey, detailing the legacy MMM system, the transition to MHA, integrations with Zebra and Proxy middleware, current challenges, and future designs such as distributed agents, semi‑sync replication, and MySQL Group Replication.
The article details Meizu's massive real‑time push system handling 25 million online users and 600 million messages per minute, explains its four‑layer architecture, and shares how the team tackled phone power consumption, mobile network instability, massive connections, monitoring, and gray‑release deployment.
The article outlines essential website architecture patterns—layered design, separation, distribution, clustering, caching, asynchronous processing, redundancy, automation, and security—explaining how each contributes to high concurrency, scalability, reliability, and maintainability of large web applications.
This article explains the design and implementation of PhxSQL, a MySQL‑compatible high‑availability solution that uses a reliable log storage based on Paxos, Proxy request forwarding, automatic master election, and other mechanisms to overcome native MySQL replication flaws and provide strong data consistency and fault‑tolerant performance.
The article outlines the progressive architectural stages of large‑scale websites—starting with a single‑server setup and advancing through service separation, caching, load balancing, database read/write splitting, CDN/reverse proxy, distributed storage, NoSQL, business splitting, and distributed services—to illustrate how high concurrency, massive traffic, high availability, and massive data are handled.
This article introduces the origin and key alliances of blockchain, explains its core characteristics such as decentralization, consensus mechanisms and transaction transparency, and provides a detailed description of the Merkle Tree algorithm, its implementation, verification steps, and applications in distributed file synchronization.
This article details Ele.me's design and implementation of a MySQL bidirectional replication component (DRC) that enables sub‑second, high‑throughput data synchronization across Beijing and Shanghai data centers, addressing latency, consistency, and failover challenges in a multi‑active environment.
This article details Taobao’s multi‑layer massive data platform, covering its five‑tier architecture, the 1500‑node Hadoop “Cloud Ladder” for batch processing, the low‑latency “Galaxy” stream engine, MySQL‑based MyFOX, HBase‑based Prom storage, the glider middle‑layer, and sophisticated caching strategies that together support petabytes of data and millions of daily queries.
This article explains the core concepts of distributed system consistency—including the CAP theorem, ACID properties, various distributed transaction techniques, and the two‑phase and three‑phase commit protocols—while illustrating practical implementations with message queues and local message tables.
This article provides an extensive overview of more than ninety open‑source and commercial big‑data tools—including ETL platforms, resource managers, storage systems, messaging queues, processing engines, and visualization libraries—detailing their core functions, typical use cases, and notable adopters.
This article describes the design, technology selection, and performance evaluation of Ctrip's real‑time user behavior data collection platform, covering Netty‑based network handling, Kafka/Hermes messaging, encryption, compression, Avro backup, and related analytics products, with detailed feasibility analysis and benchmark results.
This article explores practical techniques for synchronizing heterogeneous data sources, performing batch and incremental analytics with Hadoop and Spark, designing low‑latency real‑time computation pipelines, implementing push notifications, and choosing appropriate storage solutions—from in‑memory caches to distributed databases—while addressing performance, reliability, and scalability challenges.
This article presents a comprehensive overview of fault governance in large‑scale distributed systems, covering classic dependency failures, the concept of strong and weak dependencies, experimental observations, the evolution of fault‑injection techniques, and best practices for building reliable fault‑drill platforms.
This article consolidates e‑commerce platform architecture practices, covering design principles, multi‑level caching, indexing strategies, parallel and distributed computing, high availability, scaling techniques, resource optimization, static blueprint, component analysis, and supporting middleware such as load balancers, routers, HA, messaging, caching, buffering, search, and log collection.
This article outlines the principles and components of a scalable game server architecture, covering login management, region selection, gateway handling, communication protocols, publish‑subscribe, RPC, server merging, and the evolution from single‑threaded to cloud‑native designs.
This article explains the concept of linearizability, contrasts it with serializability, and describes how Google Spanner uses the TrueTime API and commit‑wait mechanisms to provide external consistency and reliable snapshot reads in a globally distributed database system.
PacificA is Microsoft’s generic replication framework for large‑scale distributed storage systems that provides strong consistency, separates configuration management from data replication, and uses a primary‑secondary model with lease‑based fault detection to ensure availability, correctness, and efficient operation across heterogeneous nodes.
This article traces Meituan‑Dianping's MySQL high‑availability journey from the early MMM replication manager to the modern MHA‑Zebra and MHA‑Proxy solutions, compares each architecture, highlights their shortcomings, and outlines future directions such as distributed agents, semi‑sync replication, and Paxos‑based MySQL Group Replication.
This article explains why architecture practice is essential, introduces the concept of unit (cell) architecture, discusses its performance and cost advantages, and details how Weibo applied it to its fan service platform, including handling of partitioning and job management.
The Fault‑Drill platform for hotel back‑end services combines real‑time traffic replication to shadow clusters with UI‑driven fault injection via a java‑agent, enabling developers to validate incident‑response plans, measure latency impacts, and reduce MTTR by testing normal and abnormal conditions on live traffic.
After leaving Baidu in 2015, the author recounts nearly two years of evolving a startup’s backend architecture—from a single‑server Tomcat setup to multi‑node clusters with Nginx reverse proxy, database sharding, caching, session sharing, and service‑oriented design—highlighting challenges, optimizations, and lessons learned.
This article explains why Paxos is needed for distributed consistency, describes its three roles and phases, illustrates the protocol with a real‑world analogy, and shows how Paxos‑based log replication can provide strong, fault‑tolerant high‑availability for databases.
The article discusses how expert programmers must continuously learn and balance humility with confidence, explores the end‑to‑end design principle, examines complexity, layering, and componentization in large systems, and highlights performance considerations, distributed‑system realities, and management lessons for building robust software.
This article outlines eBay's seven scalability best practices—including functional partitioning, horizontal sharding, avoiding distributed transactions, asynchronous decoupling, streaming, virtualization, and smart caching—to help large‑scale web services achieve reliable, cost‑effective growth.
This article explores core distributed system design principles, detailing the CAP theorem and its implications, the BASE extension, shared‑nothing architecture, various load‑balancing algorithms and deployment modes, as well as high‑availability strategies such as active‑standby, active‑active, and clustering to eliminate single points of failure.
This article explains how we designed and implemented a scalable OAuth token storage and refresh system using Tarantool’s in‑memory database, Raft leader election, sharding across multiple data centers, and a custom lightweight queue to handle high‑throughput token updates while maintaining consistency and fault tolerance.
In this comprehensive talk, Alibaba’s database chief Zhang Rui shares the challenges of high‑availability, cost, and elasticity in massive transaction systems, outlines innovations like AliSQL X‑Cluster, Paxos‑based consistency, X‑KV, dual‑engine storage, and discusses the evolving role of DBAs toward automation and intelligent optimization.
This article outlines the design principles and architectural strategies for constructing a high‑concurrency, high‑availability e‑commerce platform, covering space‑time tradeoffs, caching layers, indexing techniques, parallel and distributed computing, load balancing, stateless services, resource optimization, fault tolerance, data storage options, and real‑time processing components.
This article tells the story of a developer who first used a blocking email call, then switched to multithreading, and finally adopted a message queue to achieve asynchronous processing, decoupling, scalability, and reliability for user registration emails.
This article explains the concepts of distributed clusters and load balancing, contrasting clusters and distributed systems with real‑world analogies, describing various load‑balancing techniques such as DNS, LVS, and reverse proxies, and offers practical guidance on designing simple, reliable, and efficient load‑balancing solutions for distributed back‑ends.
This article explores practical techniques for handling node liveness, failover, recovery, and exactly‑once transaction semantics in distributed systems, illustrating implementations with Zookeeper, Kafka, Storm, and database sharding while addressing big‑data reach calculations and performance trade‑offs.
The HULK Container Platform scheduling system, built for Meituan‑Dianping, combines a hybrid, actor‑based scheduler with filter‑and‑rank logic, configurable trade‑offs, and dynamic over‑commit to balance resource utilization, high availability, and massive concurrent placement decisions for thousands of containerized services.
This article uses a vivid island metaphor to break down the Paxos consensus algorithm, maps its concepts to Zookeeper components such as servers, leaders, and Zxid, and illustrates conflict resolution and leader election through clear examples and diagrams.
This article chronicles the evolution of Meituan's food‑delivery order system from a simple modular prototype to a distributed, high‑performance, highly available architecture, detailing the business characteristics, architectural milestones, performance optimizations, consistency safeguards, scalability techniques, and intelligent operations that enable handling millions of orders per day.
The article introduces WeChat's self‑developed MQ 2.0 asynchronous queue, detailing its architecture, cross‑machine consumption model, improved task scheduling, efficient processing frameworks—including a MapReduce‑style engine and streaming tasks—and robust overload protection mechanisms that together boost reliability and performance for large‑scale backend services.
This article walks through the evolution of a website from a single‑server setup to a distributed, high‑performance architecture, covering service separation, caching strategies, server clustering, load balancing, database replication, CDN acceleration, distributed storage, NoSQL adoption, and modular business decomposition.
This article presents a comprehensive, Chinese‑language collection of big‑data resources—including relational databases, distributed file systems, key‑value stores, distributed programming tools, file data models, and key‑map frameworks—compiled by w3cschool to help programmers deepen their understanding of big data technologies.
This article explains the design and implementation of XDB, a MySQL high‑availability framework used by Baidu Waimai, detailing its component architecture, Horus state‑machine based failover, monitoring mechanisms, log management, and real‑world deployment experiences.
Meituan’s Leaf system merges segment‑based caching with Snowflake‑style bit fields to deliver globally unique, trend‑increasing 64‑bit IDs at ultra‑low latency, using double‑buffered DB segments, master‑slave MySQL replication, Zookeeper‑assigned worker IDs, and clock‑rollback safeguards, achieving ~50 k QPS and 1 ms 99.9th‑percentile response across billions of daily IDs.
This article explains common load‑balancing strategies—round‑robin, random, minimum response time, minimum concurrency, and hash—detailing their principles, advantages, drawbacks, and mathematical reliability analysis, including probability formulas and visual illustrations to help choose the most fault‑tolerant approach for distributed systems.
The article reviews hardware and software load‑balancing, explains classic strategies such as round‑robin, random, minimum‑response‑time, least‑connections and hash, and quantitatively evaluates their fault‑tolerance using probability formulas and example scenarios in distributed systems.
This article explains how distributed systems balance consistency, availability, and performance by covering consensus algorithms like Paxos, replication models such as Master‑Slave and Master‑Master, and Amazon Dynamo’s N‑R‑W and Vector Clock approaches, while illustrating the CAP theorem trade‑offs.
This article details how to design, implement, and benchmark a single‑machine backend capable of handling up to 1 million concurrent connections and 60 000 queries per second while simulating the shake‑and‑send red‑packet workflow of a large‑scale messaging app, including capacity calculations, architecture choices, Go‑based implementation, and multi‑stage performance testing.
This article describes how to design, implement, and evaluate a backend system that simulates the Spring Festival red‑packet service, achieving up to 6 × 10⁴ QPS on a single server while handling one million concurrent connections, and discusses the hardware, software, architecture, and performance results.
The article explains how achieving high availability through data redundancy introduces consistency challenges that in turn affect performance, and it reviews partitioning, mirroring, consistency models, replication architectures, and two/three‑phase commit protocols in distributed systems.
This article explores the challenges of mutual exclusion and idempotency in distributed environments, explains the underlying principles of locks in multi‑threaded and multi‑process contexts, and presents practical implementations using Zookeeper, Redis, Tair, and the Cerberus and GTIS frameworks to ensure reliable, scalable operations.
This article examines the challenges of achieving consistency in large‑scale distributed service systems, outlines common inconsistency scenarios such as split‑brain and lost updates, and presents practical patterns—including ACID/BASE trade‑offs, two‑phase and three‑phase commit, TCC, query, compensation, and reliable messaging—to guide engineers in designing robust, eventually consistent architectures.
This article explains the challenges of achieving consistency in distributed systems and presents practical solutions such as two‑phase commit, asynchronous assurance, compensating transactions, message retry mechanisms, idempotent designs, and a custom Redis‑based delayed queue (DelayQ) with a transactional proxy (TMQProxy) to provide reliable transactional messaging.
This article explains the key challenges of high availability and scalability in distributed architectures and provides practical solutions for each layer—entry, business, cache, and database—using techniques such as heartbeat IPs, stateless services, consistent hashing, and sharding.
The article shares Google SRE veteran Minghua Ye’s insights on App Engine’s evolution, emphasizing the critical role of automatic scalability, distributed locks, service discovery, load balancing, and open‑source tools like gRPC, Protobuf, gflags, glog, and Googletest in building reliable, high‑traffic cloud services.
The article explains how a high‑traffic e‑commerce platform like Taobao relies on distributed storage, search engines, massive caching, load‑balancing, CDN, sophisticated advertising and analytics systems, all of which require large teams of top engineers to design, implement, and operate.
The article details Alibaba's globally distributed IDC and network monitoring systems, describing their fully distributed data collection, centralized computation, storage strategies, alarm mechanisms, and frontend visualization that together enable real‑time infrastructure and network health management for large‑scale operations.
The article outlines key challenges in distributed API design—including date formatting, decimal precision, response structures, idempotency, security, and naming consistency—and provides practical recommendations to improve usability, scalability, and maintainability across backend services.
This article examines the high‑availability architectures behind massive online transaction systems such as Alibaba's Double 11 sales, Alipay and WeChat red packets, detailing the challenges of billions of requests and the engineering solutions that ensure performance, reliability, and security.
This article examines VIPshop's transformation from a single‑application LAMP system to a vertically split and finally distributed service‑oriented architecture, detailing the business model, key design requirements, platform governance, and the technical services that enable a scalable e‑commerce operation.
This article explains the theory and practical techniques behind synchronizing state across data centers using a two‑site three‑center model, covering distributed protocols like Paxos and Raft, CAP trade‑offs, latency challenges, and high‑availability designs for robust database replication.
This article explains Ceph's fault detection mechanism, detailing how OSD peers exchange heartbeats, report failures to the Monitor, and how the Monitor aggregates reports and applies configurable thresholds to reliably identify and handle downed OSD nodes in a distributed storage cluster.
This article explains the challenges of generating globally unique IDs across distributed shards, outlines the requirements for such IDs, and details Twitter's Snowflake algorithm—including its structure, generation process, and clock handling—before exploring three notable Snowflake variants and their trade‑offs.
The article provides a detailed technical overview of Scality Ring 6.0, explaining its three‑layer software stack, X86‑based scale‑out hardware, diverse connectors, storage node design, management tools, routing protocol, data durability, multi‑site deployment models, and consistency guarantees.
This article explores core system architecture concepts—including the CAP theorem and its BASE extension, the shared‑nothing design, various load‑balancing algorithms and deployment modes, and high‑availability patterns such as active‑standby, active‑active and clustering—providing practical guidance for building scalable, reliable distributed applications.
Alibaba’s transformation from e‑commerce leader to tech powerhouse is highlighted by its Feitian middleware platform, a cloud‑based, highly available solution that supports diverse industries, enables massive transaction volumes, and exemplifies the evolution of large‑scale distributed architectures pioneered since Alibaba’s early IOE days.
The article outlines the architecture, distributed design, redundancy, monitoring, automation, and fault‑handling strategies employed in a game company's platform backend to ensure high availability and efficient daily operations.
JIMDB is JD’s in‑house distributed cache platform that combines automatic fault detection, seamless online scaling, multi‑language support, and containerized deployment to replace traditional Memcached/Redis solutions, offering features such as one‑click cluster creation, elastic expansion, lossless scaling, and comprehensive monitoring for high‑traffic e‑commerce services.
This article traces the evolution of internet‑scale technical architecture across three eras—single‑machine, cluster, and distributed—detailing the motivations, core patterns, advantages, and drawbacks of monolithic, layered, data‑separated, cached, load‑balanced, CDN‑accelerated, redundant, service‑oriented, sharded, and microservice designs.
The author announces a forthcoming series on Internet high‑availability architecture, outlining topics such as CAP theory, distributed caching, SOA, message queues, search systems, and real‑world case studies, and invites readers to suggest additional content while promising detailed, valuable guidance for developers and architects.
Qcmd, the command execution system behind 360’s private HULK cloud platform, replaces SaltStack with an asynchronous, Golang‑based architecture that ensures high‑availability, encrypted messaging, and reliable mass‑host command execution across thousands of servers, dramatically reducing task timeouts and operational overhead.
This article explains what high availability means for distributed systems, outlines common availability tiers, and describes how redundancy, load balancing, and automatic failover across a typical Internet architecture can achieve reliable, scalable services.
The talk details how Alibaba tackled the massive technical challenges of Double 11 over eight years by evolving a highly available, scalable architecture through capacity planning, distributed middleware, hybrid‑cloud deployment, online stress testing, and fine‑grained traffic control to balance cost, performance, and user experience.
This article explains the Mesos distributed system kernel, its resource‑allocation workflow, and how Qunar engineers applied and evolved Mesos, Marathon, and custom frameworks to achieve fine‑grained scheduling, high availability, service discovery, and multi‑tenant management in a large‑scale production environment.
This article provides an in‑depth exploration of distributed consensus protocols, detailing the basics of Paxos, extending to Multi‑Paxos, and comparing it with Raft and Zab while discussing leader election, quorum, lease mechanisms, and practical considerations for implementing these algorithms in real‑world systems.
This article introduces core distributed‑system concepts—including the definition of consensus, the two‑phase and three‑phase commit protocols, the CAP theorem and its engineering implications, and logical‑clock mechanisms such as Lamport timestamps, vector clocks, and version vectors—explaining their models, challenges, and practical trade‑offs.
This article explains the essential design principles, trade‑offs, and core components—such as availability, performance, reliability, scalability, manageability, and cost—required to build large‑scale, high‑availability web systems and illustrates them with an image‑hosting example.
This article explains the fundamentals of transactions, the ACID properties, isolation levels, distributed transactions, the CAP theorem, and the BASE model, illustrating how they shape consistency, availability, and reliability in modern database and distributed system design.
This article outlines the key technical challenges of flash‑sale (秒杀) systems—high concurrency, cache usage, distributed locking, database pressure, and overselling—and presents practical front‑end and back‑end design patterns, including atomic counters, memcached decrements, queueing, Redis off‑loading, and two‑phase commit solutions.
This article analyzes Etcd's distributed architecture, Raft‑based consistency, storage implementation, watch and lease mechanisms, differences between v2 and v3, and compares it with Zookeeper and Consul, providing practical usage tips and surrounding tooling for developers of distributed systems.
This guide explains Git's origins, compares distributed and centralized version control, walks through creating a repository, adding and committing files, switching versions, and using reset, log, and reflog to recover previous states, providing practical command‑line examples for developers.
This article outlines Alibaba Cloud's big‑data platform challenges and solutions during the 2016 Double 11 event, covering sub‑second real‑time processing, multi‑million‑records‑per‑second throughput, full‑day high availability, and massive offline workloads exceeding hundreds of petabytes.
This article examines the fundamental challenges of achieving consistency in distributed systems, explains the CAP theorem, compares two‑phase and three‑phase commit protocols, explores XA transactions, and presents practical compensation patterns such as local message tables, non‑transactional and transactional MQ designs, highlighting their trade‑offs and applicability.
This article explains the core concepts of distributed version control, compares it with centralized systems, describes repository structures, outlines the advantages of Git, and provides step‑by‑step command examples for initializing, committing, branching, merging, cloning, pulling, and pushing changes in a collaborative development workflow.
Distributed systems rely on coordinated services such as ZooKeeper for state management, message queues like ActiveMQ for inter‑process communication, robust transaction handling, automated deployment tools like Docker, and comprehensive logging solutions, each playing a critical role in achieving high availability, scalability, and operational visibility.
This article explains the importance of data storage, describes single‑node storage engines and data models, outlines transaction and concurrency control, and covers distributed storage principles, CAP and FLP theorems, 2PC and Paxos protocols, as well as redundancy, backup, and failover mechanisms.
The article explains why Weidian needed a new big data development platform, outlines the functional features of the Mars system, describes its architecture, scheduling mechanisms, task execution flow, and discusses remaining challenges and future enhancements.
This article explains how to apply machine learning to real‑time distributed processing, covering model basics, data collection, parameter learning, topology design, message flow control, and the Hurricane framework architecture.
An in‑depth overview of NoSQL databases explains the limitations of traditional relational systems, details ACID properties, introduces the CAP theorem and BASE model, compares RDBMS with NoSQL, outlines advantages, disadvantages, history, classifications, and real‑world usage examples.
The article outlines the step‑by‑step evolution of a .NET‑based web system, describing how a single‑server setup grows into a multi‑tier, load‑balanced, clustered, stateless, micro‑service architecture that leverages caching, NoSQL, search engines, cloud services, Docker and CDN to handle large‑scale traffic and data processing.
This article examines Tencent Analytics' architecture, detailing its real-time data collection, in‑memory computation, and NoSQL storage strategies that enable second‑level updates for terabyte‑scale web traffic.
The article outlines the five‑year evolution of Qunar's payment platform from a tightly coupled monolith (1.0) to a highly available, service‑oriented distributed architecture (2.0), detailing component breakdown, challenges, and the resulting core transaction, payment, cashier, and API layers.
