The article analyzes the decline of Hadoop as the dominant big‑data platform, explains how cloud‑based services are replacing its complex on‑premises architecture, and outlines the lessons and future directions for enterprises navigating a post‑Hadoop landscape.
To guarantee stability for over 100 million daily users, Gaode combines capacity planning, traffic control, disaster recovery, monitoring, and pre‑plan drills with a self‑built full‑link load‑testing platform (TestPG) that replays realistic traffic in production‑like environments, isolates test loads, provides rapid configuration, detailed debugging, automated error capture, and comprehensive reporting, while planning future enhancements such as integrated topology monitoring, advanced pressure models, and confidence evaluation.
This article explains Elasticsearch's cluster architecture, including nodes, indices, shards, replicas, deployment models, and data layer storage, and compares two types of distributed data system designs—local file‑system based and shared‑storage based—highlighting their advantages and trade‑offs.
This article provides a comprehensive, illustrated explanation of Apache HBase's architecture, covering its master‑slave components, region management, Zookeeper coordination, data flow for reads and writes, storage structures, compaction processes, fault recovery, and the system's strengths and limitations within the Hadoop ecosystem.
An experienced Alibaba senior tech expert shares a comprehensive, step‑by‑step guide to system design, covering purpose, measurable goals, core design principles, detailed subsystem planning, and real case studies like HSF, T4, and multi‑site deployment, offering practical insights for architects to avoid common pitfalls.
At QCon Beijing 2019, senior architect Wang Junxiang shared Tong‑Cheng Yilong’s end‑to‑end degradation‑plan architecture, covering system design, data collection, metric computation, resource recovery, link‑level pre‑plan management, fault diagnosis, strategy extensibility, and high‑availability platform construction, offering practical insights for complex distributed systems.
Kafka is a distributed, partitioned, replicated messaging system originally developed by LinkedIn, offering high throughput, low latency, fault tolerance, and scalability; this article explains its core concepts, file storage design, partition replication, leader election, consumer groups, delivery guarantees, and operational considerations for big‑data pipelines.
This article introduces QMQ, Qunar's internal distributed message queue, covering its background, design motivations, core concepts, code examples for producing and consuming both real‑time and delayed messages, transactional messaging support, and the overall architecture of its metaserver, broker, and delay components.
OCTO is Meituan's distributed microservice communication and governance platform that unifies service registration, discovery, load balancing, fault tolerance, gray releases, and call data across thousands of services, handling billions of calls with components such as OCTO‑RPC, OCTO‑NS, OCTO‑Portal, SGAgent, Oceanus, Watt, and MCC, achieving 99.999% success and moving toward API‑based naming and service‑mesh evolution.
This article provides a comprehensive overview of Kafka, covering its basic concepts, architecture, deployment, configuration, monitoring, producer and consumer settings, offset management, high availability, replication, leader election, and practical tips for deployment, tuning, and troubleshooting in production environments.
FeatureKV is a high‑performance, scalable key‑value storage system built on WeChat’s internal file system and metadata service, designed to handle billion‑scale read and write demands, support batch offline writes, provide version management, and achieve low‑latency online reads for services like Look‑at‑Look, ads, and payments.
This article explains Zookeeper’s core concepts, its ZAB consensus protocol, and surveys its essential roles in major big‑data components such as Kafka, Hadoop, HBase, and Solr, illustrating how it provides configuration, naming, coordination, leader election, and high‑availability services across distributed architectures.
This article compares Apache Kafka and Apache Pulsar, detailing Kafka's scalability challenges, Pulsar's architectural benefits, performance gains, multi‑tenant support, security features, and provides code examples and migration guidance for large‑scale streaming applications.
This article demonstrates how to create Bash scripts that fully saturate CPU and I/O resources, explains their role in fault injection within the Simian Army framework, and introduces the broader concepts and benefits of chaos engineering for building resilient distributed systems.
The article explains how delayed message queues replace inefficient scheduled‑task scans in distributed systems, outlines common use cases such as order timeouts and retries, compares RabbitMQ, RocketMQ, Kafka, ActiveMQ and Redis implementations, and details NetEase’s ActiveMQ‑based solution with idempotent processing and traceability.
This article explains how 360 Search processes billions of web pages daily, detailing its backend architecture, offline indexing, online retrieval, index organization, and relevance models that enable efficient search over a hundred‑billion‑scale web corpus.
This article explains distributed data consistency concepts, the CAP theorem, various consistency levels, and provides detailed overviews of three major consensus algorithms—Paxos, ZAB, and Raft—including their mechanisms, roles, and practical applications such as in CB‑SQL.
This article explains the architecture of Elasticsearch clusters, covering core concepts such as nodes, indices, shards, replicas, indexing flow, mixed and tiered deployment models, data‑layer storage, and compares these designs with other distributed data system architectures.
This article explains why comprehensive, multi‑layer monitoring is essential for distributed systems, outlines environment, program, and business metrics, recommends practical tools such as Zabbix, open‑falcon, Prometheus and Grafana, and provides a step‑by‑step evolution plan and alerting strategy.
This article provides a comprehensive introduction to Apache Kudu, covering its origins, cluster architecture with Raft consensus, table schema and partitioning design, and detailed storage mechanisms including MemRowSet, DiskRowSet, CFile, and compaction processes.
Spring Session enables distributed session sharing by storing HTTP session data in Redis, using a filter and listener configuration to replace Tomcat’s in‑memory storage, managing three Redis keys with coordinated expirations, and subscribing to keyspace events for reliable cleanup and cross‑instance access.
This article presents thirty practical architecture principles—from keeping designs simple and avoiding unnecessary features to embracing automated testing, ROI, user‑centric decisions, and distributed system fundamentals—offering a comprehensive guide for backend developers to build scalable, maintainable, and user‑friendly software.
This document compiles extensive backend interview questions covering topics such as Java concurrency, JVM internals, design patterns, database optimization, distributed systems, Spring Cloud, Dubbo, and system design, along with promotional information for obtaining detailed answer resources.
This article walks through Taobao's architectural evolution—from a single‑server setup to distributed clusters, caching, load balancing, microservices, containerization, and finally cloud platforms—illustrating the technologies and design principles needed to handle hundred‑to‑hundred‑million concurrent requests.
This article explains the design and implementation of NSQ dual‑ and multi‑data‑center architectures using a lookup‑migrate proxy, covering deployment scenarios, routing strategies, migration phases, JSON response transformations, and practical lessons learned for reliable message publishing and consumption across data centers.
This article introduces the fundamentals of microservices and reactive systems, explains how reactive programming integrates with microservice architectures, compares traditional and reactive approaches, outlines core components, design principles, and technology choices such as Spring Cloud, and provides guidance for building scalable, resilient reactive microservices.
This article explains how Apache Flink builds its RPC communication layer on top of Akka by detailing the Actor model, actor system creation, message passing patterns, key RPC interfaces such as RpcGateway and RpcEndpoint, and the internal workflow of request handling and execution.
This article provides a comprehensive overview of Apache RocketMQ’s core architecture, including topic routing mechanisms, message storage file designs, high‑availability message sending, concurrent pull and consumption processes, HA synchronization, and transaction messaging, while offering practical learning steps and programming techniques for developers.
This article demonstrates how to build a distributed rate‑limiting component named shield‑ratelimiter using Redis’s INCR and TTL features, Spring‑Boot‑starter integration, Lua scripting for atomic operations, custom annotations, and AspectJ, providing a robust, configurable solution for limiting API calls in Java backend services.
This article explains the concept of high availability for message queues, analyzes interview expectations, and details the HA mechanisms of RabbitMQ (including single, normal cluster, and mirrored modes) and Kafka (partition replication and leader election), highlighting their advantages, drawbacks, and practical considerations.
This article walks through building an e‑commerce platform using microservices, covering module decomposition, domain‑driven design, service splitting, technology stack choices, distributed transaction strategies, circuit‑breaker patterns, centralized configuration, monitoring, and capacity planning to guide developers from concept to deployment.
This article presents a carefully organized catalogue of over a hundred technical posts covering Flink source‑code analysis, fundamental and advanced big‑data structures, Hadoop ecosystem components, real‑time streaming with Spark and Kafka, as well as system design guidelines and miscellaneous insights, each linked to its original publication for easy reference.
The article reviews Alibaba's architectural evolution from early PHP and JavaBean systems through EJB, Spring, and service‑oriented transformations using EDAS, DRDS, and ONS, highlighting challenges such as maintenance, data silos, database limits, and the resulting enterprise‑grade cloud‑native solutions.
Tencent Cloud’s database evolution progressed from early open‑source integration to self‑developed, cloud‑native solutions, producing the distributed, high‑availability TDSQL and the compute‑storage‑separated, log‑sinking CynosDB, each featuring advanced scheduling, sharding, cost‑based optimization, and asynchronous log replay to deliver scalable, low‑latency performance for diverse workloads.
The article describes Autohome's evolution from a Zabbix‑based monitoring setup to a custom, distributed monitoring platform, detailing its architectural components, design goals, implementation choices, product features, and future roadmap for fault localization and dynamic alerting.
This article explains the Paxos consensus algorithm, its purpose in solving data consistency across distributed nodes, its basic two‑phase workflow with proposer, acceptor, and learner roles, and illustrates three typical scenarios with step‑by‑step examples and code snippets.
Over two months, I interviewed with multiple companies for Java backend roles, sharing detailed experiences from technical rounds—covering JVM, concurrency, distributed locks, databases, and system design—to highlight key questions, effective answers, and practical advice for succeeding in similar backend development interviews.
This article provides a concise, ten‑minute guide to MySQL master‑slave and master‑master replication, data sharding principles and implementations, and various database deployment architectures—including single‑instance, replication‑based scaling, and sharding‑based scaling—while highlighting practical considerations, advantages, and common pitfalls.
This article explains the challenges of handling Double‑11 traffic spikes, introduces adaptive load‑balancing concepts, analyzes the 5th Middleware Performance Challenge scenario, and outlines algorithm design considerations and evaluation steps for building a robust, self‑adjusting load‑balancing solution.
The article introduces Apache Dubbo, a high‑performance lightweight Java RPC framework, outlines its core capabilities, development history, technical ecosystem, 2019 plans, shares micro‑service implementation experiences, and provides speaker and community information for the DataFun big‑data forum.
Implementing a service mesh in production faces real‑world hurdles such as significant CPU consumption, 20‑50% performance loss, tangled sidecar responsibilities, missing registration support, and control‑plane bottlenecks, which can be mitigated by a central‑mesh fallback, IPC and lock‑free optimizations, staged sidecar splitting, and unified Pilot‑based service discovery.
This article explores Hadoop’s role in the big‑data era, detailing its architecture, core components such as HDFS, YARN, MapReduce, Ozone and Submarine, the challenges of trillion‑scale data, and why its scalability, cost efficiency, and a mature ecosystem give it a competitive edge.
This article introduces Kafka, explains its core concepts and design principles, outlines typical use cases, and discusses common operational problems and troubleshooting tips for this high‑throughput distributed messaging system.
This article explains the Paxos consensus algorithm in plain terms, using a relatable travel‑planning analogy to illustrate how proposers, acceptors, and majority voting achieve fault‑tolerant agreement in distributed systems, and connects the concept to real‑world implementations like Google’s Chubby and ZooKeeper.
This article introduces RocketMQ, compares it with ActiveMQ, explains its producer/consumer group model, describes various broker deployment modes, and provides a step‑by‑step dual‑master installation guide with configuration, log, and console setup instructions.
This article presents a comprehensive overview of Taobao's cloud computing architecture, detailing system components, various data synchronization methods such as TimeTunnel, Dbsync, and DataX, the scheduling system design, and metadata-driven analysis platforms for performance optimization and monitoring.
This article presents an in‑depth interview with Alibaba Cloud senior expert Sun Tingtao, detailing the architecture, core features, design challenges, and operational strategies of the Alibaba Cloud Log Service that handles over 10 PB of daily log data for massive, diverse production workloads.
This article provides a detailed roadmap for candidates preparing for the autumn recruitment season, covering interview experience sharing, systematic learning routes, project preparation, essential Java and big‑data technologies, core algorithms, and practical interview question collections to help readers avoid common pitfalls and succeed in securing offers.
This article provides a comprehensive overview of Apache Kafka's architecture, covering topics such as producer and consumer workflows, partition and replica management, leader election, offset handling, message delivery semantics, transaction support, and file organization, illustrating how Kafka achieves high performance and scalability.
This article explores why Java remains the dominant backend language, introduces the Dubbo framework and its evolution, explains core concepts such as providers, consumers, and registries, and details practical configurations for registry, load balancing, rate limiting, governance, monitoring, and extensions like DubboX and REST support.
This article examines the rapid rise of cloud‑native databases, detailing market trends, architectural innovations such as shared storage and shared‑nothing designs, and showcasing Alibaba Cloud's POLARDB, POLARDB‑X, AnalyticDB, and autonomous tuning platform iBTune, while highlighting security and high‑availability advancements.
This article explains the CAP theorem, the trade‑offs among consistency, availability and partition tolerance, describes strong, weak and eventual consistency models, and introduces the BASE approach as a practical alternative for large‑scale distributed systems.
This article explains how to build a comprehensive monitoring system for distributed applications by classifying monitoring functions, describing data quadrants, outlining core modules such as collection, processing, feature extraction, and visualization, and reviewing typical implementations for metrics, logs, tracing, alerting, and the key open‑source components involved.
This article explains how Apache Kafka implements transactional messaging, covering the atomic write guarantees across partitions, the role of TransactionCoordinator, transaction state management, producer and consumer handling, code examples, and the end‑to‑end workflow for exactly‑once semantics.
This article walks through Redis from a basic overview to advanced features such as persistence, Sentinel, clustering, data types, transactions, Lua scripting, pipelining, and distributed locks, illustrating each concept with practical examples and explaining when and why to use them in real‑world applications.
This article presents Gong Yunfei's 2019 talk from the Beijing Apache Flink x Apache Kafka conference, detailing how ByteDance leverages Kafka for its large‑scale streaming and data processing needs.
This article summarizes the design of WeChat's massive‑scale backend, covering its evolution from early storage systems to a multi‑master PaxosStore architecture that delivers six‑nine availability, strong data consistency, rapid iteration, and a unified microservice framework for billions of daily operations.
This article explains Apache Kafka as a distributed streaming platform, detailing its key features, core APIs, topic and log architecture, partitioning, consumer groups, guarantees, and how it serves both messaging and storage roles for real‑time and batch processing in big‑data environments.
The article analyzes how HBase regions being assigned to a single RegionServer create reliability issues such as jitter, service interruptions, and data loss, examines the underlying hardware, OS, and operational factors, and proposes system optimizations and replica-based high‑availability strategies to mitigate these problems.
This article explores practical cache usage by comparing DB‑first and cache‑first approaches, discusses transaction‑based safety, cache‑aside patterns, handling high‑availability databases, local‑cache decisions, and consistency solutions among local, distributed caches and databases.
In a QCon 2019 talk, Ant Financial’s deputy CTO Hu Xi outlines the company’s fifteen‑year journey reshaping payments and micro‑loans through blockchain, AI, security, IoT and cloud computing, and details the emerging cloud‑native, high‑availability, data‑intelligent architecture that will underpin the next generation of financial technology.
Ant Group’s former CTO Hu Xi outlines the 15‑year evolution of its fintech architecture, highlighting the five BASIC technologies—blockchain, AI, security, IoT, and cloud computing—while detailing the shift to cloud‑native, distributed middleware, OceanBase, service mesh, risk‑auto‑recovery, and open‑intelligent data platforms.
This article traces the historical laws that drove computing growth, examines how Redis, MongoDB and Memcached evolved, compares client‑side, proxy and compute‑storage‑separated architectures, evaluates their trade‑offs, and answers common questions about cloud‑based distributed databases.
The article reviews Ant Financial’s fifteen‑year journey reshaping payments and micro‑loans through blockchain, AI, security, IoT and cloud computing, explains how distributed middleware, OceanBase, service‑mesh‑based cloud‑native infrastructure and open intelligent computing architectures enable high‑availability, scalable financial services, and introduces the BASIC College talent program.
This article introduces Apache Pulsar, a cloud‑native pub/sub messaging platform that solves traditional messaging system limitations with a layered architecture, offering independent scaling, multi‑tenant isolation, cross‑region replication, zero rebalancing, unified queue‑and‑stream models, and built‑in functions and proxy support.
This article explains the fundamentals of NoSQL databases, the CAP theorem trade‑offs, and why HBase—a column‑oriented, distributed NoSQL system—offers strong consistency, automatic sharding, high availability, and seamless Hadoop integration, while also outlining its ideal scenarios and limitations.
This article explains how to guarantee that order service messages are reliably delivered to RabbitMQ by using durable queues, the confirm mechanism, early persistence with Redis and scheduled retries, and idempotent processing techniques to achieve near‑100% message safety in high‑concurrency environments.
The article analyzes the evolution of technology architecture, explains the origin and purpose of the mid‑platform concept from Alibaba and JD, outlines future software trends such as AI‑driven logic, IoT UI, blockchain data, and quantum infrastructure, and proposes a layered application model for the emerging industry internet.
EasyScheduler is an open‑source big‑data workflow scheduler that uses a decentralized architecture with Master and Worker nodes coordinated via ZooKeeper, supporting DAG‑based task definitions, various task types, fault tolerance, priority handling, distributed locks, and remote logging, all illustrated with detailed component diagrams.
This article presents a hybrid sharding approach that combines range partitioning to assign ID ranges to groups and hash modulo on the total number of tables to achieve uniform data distribution while avoiding hotspots and eliminating the need for data migration during scaling.
iQIYI built a three‑component CDN Inspection System that automatically generates tasks, centrally processes and analyzes results, and runs edge measurements to monitor millions of hybrid CDN servers in real time, detecting configuration errors, file mismatches and traffic anomalies, enabling proactive remediation and 100 % local coverage.
This article walks through the fundamentals of search engines, explaining inverted indexes, the explosion of index size, core Elasticsearch concepts, its distributed architecture, and how it powers the ELK stack for log analysis, all illustrated with clear diagrams and examples.
YouZan’s order data synchronization moves changes from MySQL through Canal‑parsed binlogs into a message queue, then uses sequential SeqNo‑based optimistic locking and HBase’s column‑version timestamps to guarantee ordering for both single‑ and multi‑table updates, while a Logstash‑style configurable pipeline feeds ES for search and HBase for detail queries, eliminating ordered‑queue bottlenecks and ensuring high‑throughput consistency.
The article explains the CAP theorem, detailing how distributed systems must trade off consistency, availability, and partition tolerance, and explores CP vs AP designs, then introduces the BASE model—Basically Available, Soft State, Eventual Consistency—as a practical complement for real‑world architectures.
This article details Youzan's evolution from a simple Flume‑based log collector to a multi‑tenant, Kafka‑buffered, Spark‑processed, HBase‑backed logging architecture that now handles hundreds of billions of log entries per day, highlighting challenges, design decisions, and future improvements.
This article outlines essential microservice principles, from business‑centric service design and standardised interaction protocols to practical splitting, aggregation, technology stack selection, API design, unified logging, error handling and legacy challenges, offering a comprehensive roadmap for successful backend service implementation.
This article explains the fundamental concepts of blockchain, outlines its three-layer architecture (protocol, extension, application), reviews typical language‑specific implementations, and presents a knowledge‑map that helps developers systematically study and build blockchain‑based products.
This article explores how to achieve high cohesion and low coupling in distributed systems by examining clean, hexagonal, and CQRS architectures, detailing domain‑driven design layers, microservice boundary design, splitting principles, and practical strategies for data, caching, and resilience in modern backend development.
This article explains the evolution from traditional to modern instant‑messaging system architectures, introduces a Timeline logical model for message sync and storage, discusses read‑ and write‑diffusion strategies, evaluates database requirements, and demonstrates a TableStore‑based implementation with sample code.
This article explains the fundamentals of database transactions, the ACID properties, how they differ in distributed environments, and details two‑phase and three‑phase commit protocols while offering personal insights on the challenges of implementing distributed transactions.
This article examines a distributed system concurrency problem, compares lock‑based transactions with multiversion concurrency control (MVCC), and explains why MVCC often outperforms pessimistic locking in scenarios demanding high read responsiveness and low contention.
Load balancing distributes traffic across multiple servers to improve performance, ensure high availability, and enable scalability, covering concepts such as vertical and horizontal scaling, business partitioning, various load‑balancing methods (DNS, HTTP, IP, layer‑2, hybrid), and common algorithms like round‑robin, random, least connections, hash, and weighted distribution.
ChaosBlade is an open‑source chaos engineering tool from Alibaba that lets you repeatedly inject failures into distributed systems, helping you measure fault tolerance, validate orchestration, test platform robustness, verify monitoring alerts, and improve emergency response capabilities for more reliable cloud‑native applications.
This article examines the challenges of handling millions of concurrent QQGame players, explains why client‑side room‑count replication is essential, and proposes a divide‑and‑conquer, scale‑out server cluster with autonomous room and region managers to achieve high availability and data consistency.
This article explains why RPC is essential for modern distributed systems, defines its core concepts, outlines the typical architecture and call flow, compares it with REST/SOAP/SOA, and reviews key technologies and popular Java RPC frameworks.
This comprehensive guide explains why load balancing is essential for high‑traffic websites, details vertical and horizontal scaling, compares DNS, IP, link‑layer, and hybrid approaches, outlines common algorithms such as round‑robin and weighted, and reviews hardware versus software solutions.
This article reviews the shortcomings of MyCat in handling sharding, joins, inserts, and session variables, demonstrates how the DBLE middleware addresses those issues with better correctness, performance, security, and operational management, and discusses code‑quality improvements and automated testing practices.
This article explains what Dubbo is, its key features such as transparent RPC, cluster fault tolerance, communication protocols, registration centers, serialization options, configuration details, load‑balancing strategies, security mechanisms, common issues and solutions, and compares it with Dubbox and other distributed frameworks.
The author shares a personal journey from multiple Alibaba interview rejections to mastering backend engineering through diligent source‑code study, open‑source contributions, algorithm training, and practical project experience, offering actionable advice for aspiring developers seeking growth and interview success.
This article provides a comprehensive technical guide on HBase, covering its core concepts, advantages and drawbacks, architecture layers, practical use cases, and a detailed step‑by‑step process for large‑scale cross‑datacenter migration using snapshot‑based strategies, with commands, diagrams, and lessons learned.
The article chronicles the design, implementation, and iterative evolution of JD Digital Technology’s in‑house host monitoring platform Diting, detailing its V1, V2, and V3 architectures, the challenges encountered at each stage, and future directions toward intelligent, automated operations.
This article explains the inner workings of the Kubernetes scheduler, covering its architecture, pod queue handling, filtering, prioritization, binding, preemption, and code-level details, while also discussing current limitations and future enhancements such as the V2 framework and gang scheduling extensions.
This article explains Redis as an in‑memory data store used for caching, database, and messaging, walks through its evolution from simple HTTP caching to dedicated servers, and details server‑side features like persistence, Sentinel, replication, clustering, as well as client‑side capabilities such as rich data types, transactions, Lua scripting, pipelining, and distributed locks.
This article provides a comprehensive overview of NSQ, covering its core components, message flow, producer and consumer interactions, key configuration parameters, handling of timeouts, backoff mechanisms, service discovery with nsqlookupd, and practical considerations for scaling and reliability.
Alibaba’s MonkeyKing, inspired by Netflix’s Chaos Monkey, employs intentional fault injection—from random node kills to simulated network outages—to test and improve system robustness across IaaS, PaaS, and SaaS layers, offering a comprehensive model for reliability engineering in complex distributed environments.
This article surveys thirty major blockchain and DAG consensus algorithms, detailing each method's advantages, disadvantages, typical users, and classification, while explaining how these mechanisms underpin decentralized trust and security across distributed ledger platforms.
This article presents a curated list of the most popular Java open‑source projects on GitHub for January, summarizing each repository’s purpose, key features, and star count to help developers discover valuable resources for learning, building, and advancing Java applications.
The article first outlines HBase’s overall architecture and core components, then details the end‑to‑end read path—from client request routing to RegionServer processing, data organization and filtering—and finally presents practical client‑ and server‑side optimizations such as heterogeneous storage, HDFS short‑circuit, hedged reads, high‑availability reads, and warm‑up failure fixes, illustrated with Youzan’s production cluster.
This article introduces the "Big Data Mastery" series and outlines practical solutions for three frequent distributed system challenges—distributed locks, distributed transactions, and distributed ID generation—while highlighting open‑source tools and key design requirements.
This article explains why rate limiting is essential for high‑availability systems, compares single‑node and distributed approaches, and provides a complete Redis + Lua implementation with Java integration, including code samples and architectural diagrams to help developers.
