This article introduces the fundamental concepts of Git for enterprise developers, covering its distributed version control model, core operations such as commits, branches, and file states, and highlighting its advantages like parallel development, faster releases, strong community support, and integration with modern tooling.
This article explains the fundamentals of database transactions, the ACID properties, and how they are implemented in relational systems before exploring distributed transaction challenges and solutions such as 2PC, 3PC, TCC, and message‑based approaches.
This article explains the concept of load balancing in distributed systems, outlines its benefits for throughput and reliability, compares common architectural layers, evaluates key algorithmic considerations, and provides Python implementations of round‑robin, weighted, random, hash‑based, and least‑connection strategies along with deployment options.
This article explains why and when to move to distributed architecture, outlines the typical upgrade and splitting steps, and details five common distributed cluster patterns—including load balancing, leader election, blockchain, master‑slave, and consistent hashing—highlighting their trade‑offs and use cases.
This article details the implementation of a custom circuit breaker to prevent system failures in distributed systems, covering design principles, Java and Python implementations, and its effectiveness during high traffic periods.
This article explains the relationships among Storm's core components, how parallelism is calculated from configuration parameters, and how Nimbus performs task assignment and load balancing across workers, supervisors, and executors in a distributed streaming topology.
The article reviews Alibaba’s 2017 open‑source contributions, highlighting nine major projects—including Dubbo, RocketMQ, Druid, Fastjson, ApsaraCache, Pouch, Dragonfly, Ant Design, and Egg—detailing their features, community impact, and how they advance high‑performance distributed systems, cloud‑native computing, and modern application development.
This article explores how distributed systems determine node liveness, manage failover and recovery, and implement at‑most‑once, at‑least‑once, and exactly‑once processing guarantees—including opaque transactions and two‑phase commit—using examples from Kafka, Zookeeper, and big‑data pipelines.
Alibaba’s database team explains how their next‑generation X‑DB system achieved extreme elasticity, high performance, and cost efficiency during the Double 11 shopping festival by leveraging cloud‑native hybrid deployment, containerization, storage‑compute separation, Paxos‑based consistency, and AI‑driven self‑optimizing DBA tools, while outlining key challenges and solutions.
This article outlines practical strategies for building and scaling web applications—covering caching, static page generation, database optimization, read/write separation, NoSQL, Hadoop, distributed deployment, service separation, and CDN—to handle massive data volumes and high‑traffic loads efficiently.
This article details the business scope, challenges, architectural evolution, and practical solutions of Suning's inventory system—including front‑mid‑back separation, self‑developed high‑concurrency services, unitization, multi‑active deployment, and pre‑Double 11 capacity planning—to ensure stable, scalable e‑commerce operations during massive traffic spikes.
The article explores Twitter’s Manhattan storage system, detailing its architecture, CAP trade‑offs across various database types, the design of its modular storage engines, high‑performance operations, and the DevOps practices that enable reliable, low‑latency handling of billions of requests in a massive distributed environment.
This article explains the fundamentals of relational database transactions, the ACID properties, and various distributed transaction protocols such as 2PC, 3PC, TCC, message‑based approaches, and 1PC, while discussing their advantages, drawbacks, and practical considerations.
This article explains Ctrip's design of a globally unique, high‑concurrency user ID generator for sharded MySQL databases, reviews common industry solutions, and details the final optimized approach that combines a MySQL auto‑increment table with in‑memory segment allocation to achieve millisecond‑level response times.
This article explains the architecture and operation of Qcmd‑HTTP, a Golang‑based task distribution system on the 360 cloud platform, covering its high‑availability cluster design, task types, execution flow, status tracking, and callback mechanisms for decoupled backend services.
This article introduces RabbitMQ's role in distributed systems, explains the AMQP protocol, provides step‑by‑step Ubuntu installation commands, and details essential objects such as messages, producers, consumers, queues, exchanges, bindings, routing keys, and exchange types.
Facebook’s journey from early startup ops to supporting over 2 billion monthly users reveals how disaggregated network, storage, and warm‑storage‑enabled Spark architectures overcome scalability bottlenecks, illustrating the operational strategies and design principles that power massive, reliable data‑center services.
This article explains how HBase achieves scalable region auto‑splitting, detailing the various split policies, the algorithm for locating split points, the transactional split workflow, reference file handling, data migration via compaction, cleanup procedures, and common troubleshooting tips.
This article explains why rate limiting is essential for high‑traffic services, describes the token‑bucket algorithm, shows how to use Guava's RateLimiter and Cache for single‑node limits, and presents a Redis‑based solution that works across distributed instances.
This article explains what IRC is and outlines eight technical advantages—distributed architecture, low bandwidth, privacy, cross‑platform support, and a rich bot ecosystem—plus additional cultural reasons why many open‑source developers continue to favor this decades‑old chat protocol.
This article outlines the shortcomings of using crontab for large‑scale job execution, defines the requirements for a custom distributed scheduler, describes its three‑component architecture (trigger, monitor, management), and details key technical solutions such as process isolation, distributed locking, and log aggregation.
This article chronicles the evolution of WeChat Pay’s transaction record system—from early key/value storage bottlenecks and incomplete data to a distributed, tiered architecture that supports billions of daily records, improves query performance, ensures data security, and handles holiday traffic spikes through flexible throttling.
This article examines Uber's rapid 38‑fold growth by detailing the design, scaling techniques, and fault‑tolerance mechanisms of its real‑time market platform, including geographic indexing, microservices, distributed storage, and the DISCO scheduling system.
The article explains how traditional microservice architectures evolve into event‑driven reactive microsystems by adopting events‑first DDD, reactive design, and event‑based persistence, highlighting the role of the Actor model, asynchronous non‑blocking communication, event sourcing, and saga‑based distributed transaction handling.
This article explains how web crawlers work—from initial URL seeding and request handling to flow control, content extraction, and handling dynamic pages—while covering essential modules, HTTP details, common obstacles like JavaScript rendering, anti‑scraping measures, and strategies for large‑scale, distributed crawling.
The article outlines core guiding principles, high‑availability strategies, and high‑concurrency techniques—such as stateless design, replica and isolation, quota control, monitoring, degradation, rollback, and scaling—to help engineers build resilient, scalable web architectures for massive traffic.
This article examines the requirements for globally unique IDs in distributed systems, compares classic generation schemes such as UUID, Flickr, Snowflake and TDDL, and details a customized Snowflake‑based implementation with ZooKeeper‑managed worker IDs, clock‑rollback handling, deployment optimizations, and JVM tuning to achieve high performance and reliability.
This article explains how Tencent's ZhiYun full‑link log monitoring platform handles massive daily logs, overcomes challenges of diverse log formats, high throughput, fault‑tolerant design, and provides scalable storage, query, and alerting capabilities for distributed micro‑service environments.
This article details the background, architectural evolution, design challenges, and distributed implementation of an accounting system that automates the processing of millions of transaction records across thousands of accounts, highlighting how splitting accounts, workflows, and bills improves performance and reliability.
This article details JD.com's multi-active distributed architecture, covering its evolution from single‑data‑center to multi‑region deployments, network design, leaf‑spine topology, data consistency mechanisms, application scheduling, monitoring, and disaster recovery strategies that enhance high availability and user experience.
The article details the Jingmai Message Center’s end‑to‑end architecture, covering message ingestion via Anycall and MQ, protocol conversion, Netty‑based push system, Snowflake ID generation, Elasticsearch storage, multi‑level caching, distributed locking, and the overall design principles that enable a scalable, reliable messaging platform.
Learn how early monolithic websites evolve into distributed architectures by splitting applications, services, and data, implementing load balancers, reverse proxies, caching, CDN, database sharding, and security measures, while focusing on performance, high availability, scalability, and extensibility for robust, high‑traffic sites.
The article explains how NetEase’s real‑time big‑data stream computing platform, Sloth, handles massive, continuously generated data during China’s Double 11 shopping festival, covering use cases, architectural shifts from batch to incremental processing, technical challenges, and the role of stream‑SQL for easier development.
The article explains how continuous data generation and real‑time stream processing enable e‑commerce platforms like NetEase Kaola to handle massive Double 11 traffic, showcasing use cases, architectural shifts from batch to incremental computing, and the technical challenges of latency, accuracy, and fault tolerance.
The article explains how distributed systems achieve linearizability by using logical clocks, TrueTime, Hybrid Logical Clocks, and Timestamp Oracle mechanisms to establish a global ordering of events despite clock skew and network delays.
This article presents a collection of Alipay’s system architecture diagrams—including settlement, customer service, processing, funds, and finance components—providing a reference view of the payment platform’s core backend structure, which remains largely unchanged despite data age.
This article summarizes key insights from the Redis track at the Cloud Xi Conference, covering Alibaba Cloud ApsaraCache's unique features, Redis Enterprise's market dominance and modules, Codis's evolution and asynchronous migration techniques, and Redisson's advanced Java client capabilities for distributed caching and locking.
A senior Alibaba engineer reflects on three years of building a large‑scale data platform, tackling distributed rate‑limiting challenges, leading cross‑regional projects, and pursuing AI research, while sharing personal insights on career growth, technical problem‑solving, and the value of continuous learning.
This article outlines Taobao's learning objectives, traces the evolution of its backend architecture from V1.0 to V3.0, highlights the technical challenges faced at each stage, and explains the architectural decisions—such as modularization, service‑oriented frameworks, distributed storage, and large‑scale monitoring—that enabled massive scalability, reliability, and performance improvements.
This article traces the evolution of relational databases, explains the rise of cloud‑native computing, and details how Alibaba Cloud’s PolarDB combines storage‑compute separation, RDMA networking, shared‑disk architecture, and advanced replication techniques to deliver high‑performance, scalable, and cost‑effective database services.
This article examines various consistency solutions for distributed systems, including strong and eventual consistency, local database transactions, two‑phase commit, TCC, rollback mechanisms, local message tables, and compensation techniques, illustrating their trade‑offs and appropriate application scenarios.
This article explores the principles and components of large‑scale internet architecture, covering goals such as low cost, high performance, availability and scalability, and detailing practical implementations of DNS, CDN, load balancing, web services, caching, proxies, indexing, and queueing to build robust, efficient systems.
Apache RocketMQ, Alibaba's high‑performance distributed messaging middleware, has officially graduated to an Apache Top‑Level Project, highlighting its robust features, broad ecosystem adoption across e‑commerce, IoT, finance and big‑data, and Alibaba's strong commitment to open‑source leadership.
The article explains how RDMA (Remote Direct Memory Access) reduces CPU involvement, lowers latency, and increases bandwidth in data‑center networks, describes JD.com’s practical deployments across AI, big‑data, storage, and HPC workloads, and highlights industry trends toward broader RDMA adoption.
This article traces the history of Linux's source‑code management, explains why Linus Torvalds created Git as a distributed version‑control system, compares it with centralized systems like CVS and SVN, and highlights Git’s key advantages and ecosystem.
This article explains why Kafka can be used for permanent data retention, outlines practical use cases such as event logging, cache rebuilding, stream recomputation, and database change capture, and clarifies why Kafka is a stream‑processing platform rather than a traditional message queue or database.
PhxQueue, an open‑source, Paxos‑based distributed queue from WeChat, delivers at‑least‑once delivery, synchronous disk flushing, strict ordering, multi‑subscription, and high availability, outperforming Kafka in reliability and latency while maintaining comparable throughput, as demonstrated through detailed design, performance, and failover analyses.
This article recounts Xiaomi's rapid development of a high‑concurrency flash‑sale platform for the 2014 "Mi Fan Festival", detailing the design choices, trade‑offs, and performance optimizations that enabled millions of users to purchase phones simultaneously.
The article presents the architecture, component design, optimizations, and reliability guarantees of Bilibili's Lancer log collection system, a Flume‑based distributed pipeline that handles both real‑time and offline data streams for billions of events daily.
This article outlines the progressive stages of system architecture—from a single‑server LAMP setup through application‑data separation, caching, clustering, read/write splitting, CDN, distributed storage, NoSQL, business decomposition, and finally distributed services—while detailing essential technologies such as message queues, service frameworks, service buses, communication patterns, and governance mechanisms like Dubbo and OSB.
This article explains how a web system can evolve from handling 100,000 daily visits to over 100 million by progressively implementing multi‑level caching, various load‑balancing techniques—including HTTP redirects, reverse‑proxy, IP‑level, DNS, and GSLB—and optimizing MySQL through indexing, connection pooling, sharding, replication, and integrating memory caches such as Redis to ensure high performance and reliability.
This article explores the core concepts, design patterns, and practical techniques—including caching, distributed coordination, front‑end scaling, and back‑end queuing—to build robust high‑concurrency flash‑sale systems while preventing oversell and performance bottlenecks.
In this talk, chief architect Li Yanpeng shares his career background, outlines the goals and design principles of distributed service architecture—including high availability, performance, scalability, extensibility, security, and consistency—and offers a methodology for cultivating both technical and personal inner skills for engineers.
This article presents a comprehensive design for a high‑traffic insurance O2O platform, detailing functional requirements, system analysis, storage and caching strategies, logical and service architectures, distributed transaction handling, and the chosen development stack, emphasizing simplicity, scalability, and high concurrency.
This article dissects the technical challenges of building a high‑concurrency flash‑sale (seckill) system—covering business analysis, traffic isolation, static page caching, CDN bandwidth, dynamic order URLs, request throttling, database sharding, optimistic locking, and anti‑cheat mechanisms—while presenting concrete architectural principles and code examples.
The article provides a comprehensive overview of Ceph, covering its object storage concepts, architecture, core technologies, performance testing results versus Swift, and concludes with a summary of findings and future outlook for this distributed storage solution.
This article reviews the evolution of Meituan‑Dianping's MySQL high‑availability architecture over recent years, detailing the transition from the MMM replication manager to MHA, the integration of Zebra and Proxy middleware, and future design considerations such as distributed agents, semi‑synchronous replication, and MySQL Group Replication.
This article traces Meituan‑Dianping's MySQL high‑availability journey, detailing the legacy MMM system, the transition to MHA, integrations with Zebra and Proxy middleware, current challenges, and future designs such as distributed agents, semi‑sync replication, and MySQL Group Replication.
The article details Meizu's massive real‑time push system handling 25 million online users and 600 million messages per minute, explains its four‑layer architecture, and shares how the team tackled phone power consumption, mobile network instability, massive connections, monitoring, and gray‑release deployment.
The article outlines essential website architecture patterns—layered design, separation, distribution, clustering, caching, asynchronous processing, redundancy, automation, and security—explaining how each contributes to high concurrency, scalability, reliability, and maintainability of large web applications.
This article explains the design and implementation of PhxSQL, a MySQL‑compatible high‑availability solution that uses a reliable log storage based on Paxos, Proxy request forwarding, automatic master election, and other mechanisms to overcome native MySQL replication flaws and provide strong data consistency and fault‑tolerant performance.
The article outlines the progressive architectural stages of large‑scale websites—starting with a single‑server setup and advancing through service separation, caching, load balancing, database read/write splitting, CDN/reverse proxy, distributed storage, NoSQL, business splitting, and distributed services—to illustrate how high concurrency, massive traffic, high availability, and massive data are handled.
This article introduces the origin and key alliances of blockchain, explains its core characteristics such as decentralization, consensus mechanisms and transaction transparency, and provides a detailed description of the Merkle Tree algorithm, its implementation, verification steps, and applications in distributed file synchronization.
This article details Ele.me's design and implementation of a MySQL bidirectional replication component (DRC) that enables sub‑second, high‑throughput data synchronization across Beijing and Shanghai data centers, addressing latency, consistency, and failover challenges in a multi‑active environment.
This article details Taobao’s multi‑layer massive data platform, covering its five‑tier architecture, the 1500‑node Hadoop “Cloud Ladder” for batch processing, the low‑latency “Galaxy” stream engine, MySQL‑based MyFOX, HBase‑based Prom storage, the glider middle‑layer, and sophisticated caching strategies that together support petabytes of data and millions of daily queries.
This article explains the core concepts of distributed system consistency—including the CAP theorem, ACID properties, various distributed transaction techniques, and the two‑phase and three‑phase commit protocols—while illustrating practical implementations with message queues and local message tables.
This article provides an extensive overview of more than ninety open‑source and commercial big‑data tools—including ETL platforms, resource managers, storage systems, messaging queues, processing engines, and visualization libraries—detailing their core functions, typical use cases, and notable adopters.
This article describes the design, technology selection, and performance evaluation of Ctrip's real‑time user behavior data collection platform, covering Netty‑based network handling, Kafka/Hermes messaging, encryption, compression, Avro backup, and related analytics products, with detailed feasibility analysis and benchmark results.
This article explores practical techniques for synchronizing heterogeneous data sources, performing batch and incremental analytics with Hadoop and Spark, designing low‑latency real‑time computation pipelines, implementing push notifications, and choosing appropriate storage solutions—from in‑memory caches to distributed databases—while addressing performance, reliability, and scalability challenges.
This article presents a comprehensive overview of fault governance in large‑scale distributed systems, covering classic dependency failures, the concept of strong and weak dependencies, experimental observations, the evolution of fault‑injection techniques, and best practices for building reliable fault‑drill platforms.
This article consolidates e‑commerce platform architecture practices, covering design principles, multi‑level caching, indexing strategies, parallel and distributed computing, high availability, scaling techniques, resource optimization, static blueprint, component analysis, and supporting middleware such as load balancers, routers, HA, messaging, caching, buffering, search, and log collection.
This article outlines the principles and components of a scalable game server architecture, covering login management, region selection, gateway handling, communication protocols, publish‑subscribe, RPC, server merging, and the evolution from single‑threaded to cloud‑native designs.
This article explains the concept of linearizability, contrasts it with serializability, and describes how Google Spanner uses the TrueTime API and commit‑wait mechanisms to provide external consistency and reliable snapshot reads in a globally distributed database system.
PacificA is Microsoft’s generic replication framework for large‑scale distributed storage systems that provides strong consistency, separates configuration management from data replication, and uses a primary‑secondary model with lease‑based fault detection to ensure availability, correctness, and efficient operation across heterogeneous nodes.
This article traces Meituan‑Dianping's MySQL high‑availability journey from the early MMM replication manager to the modern MHA‑Zebra and MHA‑Proxy solutions, compares each architecture, highlights their shortcomings, and outlines future directions such as distributed agents, semi‑sync replication, and Paxos‑based MySQL Group Replication.
This article explains why architecture practice is essential, introduces the concept of unit (cell) architecture, discusses its performance and cost advantages, and details how Weibo applied it to its fan service platform, including handling of partitioning and job management.
The Fault‑Drill platform for hotel back‑end services combines real‑time traffic replication to shadow clusters with UI‑driven fault injection via a java‑agent, enabling developers to validate incident‑response plans, measure latency impacts, and reduce MTTR by testing normal and abnormal conditions on live traffic.
After leaving Baidu in 2015, the author recounts nearly two years of evolving a startup’s backend architecture—from a single‑server Tomcat setup to multi‑node clusters with Nginx reverse proxy, database sharding, caching, session sharing, and service‑oriented design—highlighting challenges, optimizations, and lessons learned.
This article explains why Paxos is needed for distributed consistency, describes its three roles and phases, illustrates the protocol with a real‑world analogy, and shows how Paxos‑based log replication can provide strong, fault‑tolerant high‑availability for databases.
The article discusses how expert programmers must continuously learn and balance humility with confidence, explores the end‑to‑end design principle, examines complexity, layering, and componentization in large systems, and highlights performance considerations, distributed‑system realities, and management lessons for building robust software.
This article outlines eBay's seven scalability best practices—including functional partitioning, horizontal sharding, avoiding distributed transactions, asynchronous decoupling, streaming, virtualization, and smart caching—to help large‑scale web services achieve reliable, cost‑effective growth.
This article explores core distributed system design principles, detailing the CAP theorem and its implications, the BASE extension, shared‑nothing architecture, various load‑balancing algorithms and deployment modes, as well as high‑availability strategies such as active‑standby, active‑active, and clustering to eliminate single points of failure.
This article explains how we designed and implemented a scalable OAuth token storage and refresh system using Tarantool’s in‑memory database, Raft leader election, sharding across multiple data centers, and a custom lightweight queue to handle high‑throughput token updates while maintaining consistency and fault tolerance.
In this comprehensive talk, Alibaba’s database chief Zhang Rui shares the challenges of high‑availability, cost, and elasticity in massive transaction systems, outlines innovations like AliSQL X‑Cluster, Paxos‑based consistency, X‑KV, dual‑engine storage, and discusses the evolving role of DBAs toward automation and intelligent optimization.
This article outlines the design principles and architectural strategies for constructing a high‑concurrency, high‑availability e‑commerce platform, covering space‑time tradeoffs, caching layers, indexing techniques, parallel and distributed computing, load balancing, stateless services, resource optimization, fault tolerance, data storage options, and real‑time processing components.
This article tells the story of a developer who first used a blocking email call, then switched to multithreading, and finally adopted a message queue to achieve asynchronous processing, decoupling, scalability, and reliability for user registration emails.
This article explains the concepts of distributed clusters and load balancing, contrasting clusters and distributed systems with real‑world analogies, describing various load‑balancing techniques such as DNS, LVS, and reverse proxies, and offers practical guidance on designing simple, reliable, and efficient load‑balancing solutions for distributed back‑ends.
This article explores practical techniques for handling node liveness, failover, recovery, and exactly‑once transaction semantics in distributed systems, illustrating implementations with Zookeeper, Kafka, Storm, and database sharding while addressing big‑data reach calculations and performance trade‑offs.
The HULK Container Platform scheduling system, built for Meituan‑Dianping, combines a hybrid, actor‑based scheduler with filter‑and‑rank logic, configurable trade‑offs, and dynamic over‑commit to balance resource utilization, high availability, and massive concurrent placement decisions for thousands of containerized services.
This article uses a vivid island metaphor to break down the Paxos consensus algorithm, maps its concepts to Zookeeper components such as servers, leaders, and Zxid, and illustrates conflict resolution and leader election through clear examples and diagrams.
This article chronicles the evolution of Meituan's food‑delivery order system from a simple modular prototype to a distributed, high‑performance, highly available architecture, detailing the business characteristics, architectural milestones, performance optimizations, consistency safeguards, scalability techniques, and intelligent operations that enable handling millions of orders per day.
The article introduces WeChat's self‑developed MQ 2.0 asynchronous queue, detailing its architecture, cross‑machine consumption model, improved task scheduling, efficient processing frameworks—including a MapReduce‑style engine and streaming tasks—and robust overload protection mechanisms that together boost reliability and performance for large‑scale backend services.
This article walks through the evolution of a website from a single‑server setup to a distributed, high‑performance architecture, covering service separation, caching strategies, server clustering, load balancing, database replication, CDN acceleration, distributed storage, NoSQL adoption, and modular business decomposition.
This article presents a comprehensive, Chinese‑language collection of big‑data resources—including relational databases, distributed file systems, key‑value stores, distributed programming tools, file data models, and key‑map frameworks—compiled by w3cschool to help programmers deepen their understanding of big data technologies.
This article explains the design and implementation of XDB, a MySQL high‑availability framework used by Baidu Waimai, detailing its component architecture, Horus state‑machine based failover, monitoring mechanisms, log management, and real‑world deployment experiences.
Meituan’s Leaf system merges segment‑based caching with Snowflake‑style bit fields to deliver globally unique, trend‑increasing 64‑bit IDs at ultra‑low latency, using double‑buffered DB segments, master‑slave MySQL replication, Zookeeper‑assigned worker IDs, and clock‑rollback safeguards, achieving ~50 k QPS and 1 ms 99.9th‑percentile response across billions of daily IDs.
This article explains common load‑balancing strategies—round‑robin, random, minimum response time, minimum concurrency, and hash—detailing their principles, advantages, drawbacks, and mathematical reliability analysis, including probability formulas and visual illustrations to help choose the most fault‑tolerant approach for distributed systems.
The article reviews hardware and software load‑balancing, explains classic strategies such as round‑robin, random, minimum‑response‑time, least‑connections and hash, and quantitatively evaluates their fault‑tolerance using probability formulas and example scenarios in distributed systems.
This article explains how distributed systems balance consistency, availability, and performance by covering consensus algorithms like Paxos, replication models such as Master‑Slave and Master‑Master, and Amazon Dynamo’s N‑R‑W and Vector Clock approaches, while illustrating the CAP theorem trade‑offs.
This article details how to design, implement, and benchmark a single‑machine backend capable of handling up to 1 million concurrent connections and 60 000 queries per second while simulating the shake‑and‑send red‑packet workflow of a large‑scale messaging app, including capacity calculations, architecture choices, Go‑based implementation, and multi‑stage performance testing.
