This article provides a comprehensive technical analysis of Kafka’s Controller component, covering its background, core responsibilities, data storage, election process, version‑specific improvements, monitoring techniques, and key source‑code excerpts to help engineers understand and manage Kafka clusters effectively.
This article shares a collection of powerful kubectl commands and tips—including API debugging, status‑based pod filtering and deletion, node‑specific pod listing, distribution counting with awk, and proxy usage—to help experienced Kubernetes users work more efficiently and avoid manual API client coding.
This article extends previous Elasticsearch cheat sheets with practical commands and step‑by‑step instructions for shard allocation, replica adjustment, cluster settings, slow‑log configuration, mapping routing, force merge, bulk writes, refresh intervals, translog durability, heap sizing, disk‑space monitoring, and troubleshooting strategies.
This article provides a comprehensive overview of Elasticsearch, covering its fundamental architecture, key concepts such as indices, shards and replicas, the complete write and search workflows, consistency mechanisms, master node election, and practical performance‑tuning recommendations for large‑scale deployments.
The first Shenzhen Elasticsearch meetup on August 21, 2021, jointly hosted by the ES Chinese community and Tencent Cloud, gathered experts from Tencent, Tapdata, ByteDance and Vivo to showcase rapid community growth, compression‑encoding optimizations, real‑time ES‑MongoDB data fusion, custom kernel extensions, large‑scale cluster practices, and concluded with extensive Q&A and networking.
This article explains the split‑brain phenomenon in distributed clusters, uses Zookeeper as a case study to illustrate how network partitions can create multiple leaders, and details Zookeeper's majority‑quorum mechanism, node count considerations, and common strategies for avoiding split‑brain scenarios.
The article examines Kubernetes' widespread adoption, outlines its scalability and multi‑cluster management constraints, discusses practical application scenarios such as deployment models, batch scheduling, and hard multi‑tenancy, and highlights the gaps that still limit its use in large‑scale production environments.
This article details JD's OLAP implementation using ClickHouse as the primary engine and Doris as a secondary engine, covering business scenarios, selection criteria, multi‑tenant deployment, high‑availability architecture, encountered challenges, and future roadmap for cloud‑native, scalable analytics.
This article details 58.com’s journey with Elasticsearch, covering the challenges of disparate deployments, common problems like disk exhaustion and write slowdown, the governance and automation platform they built, development standards, service architecture, real‑world application cases, and future plans for version upgrades and intelligent diagnostics.
Effective traffic governance and protection rely on properly configuring thresholds—either a fixed global limit that stays constant regardless of node count, or a per‑machine allocation that scales the total capacity as nodes are added or removed—to prevent sudden surges from overwhelming services and ensure high availability.
This article provides a comprehensive overview of Zookeeper's cluster deployment, explains the four server states, details the leader election process—including initialization, voting, and decision logic—and presents key source code snippets to help developers understand and implement Zookeeper's high‑availability mechanisms.
The article introduces JD Retail's Jiushu Business Analytics Platform, detailing how AI, big‑data, and distributed‑training technologies address fragmented retail scenarios, high deployment barriers, large‑scale application difficulties, and cost concerns through specialized frameworks, fault‑tolerant training, and advanced cluster optimization.
This article explains how Facebook’s Twine system orchestrates containers across millions of servers, detailing its architecture, support for stateful services, cross‑data‑center control, elastic capacity handling, and the lessons learned from eight years of large‑scale operations.
This guide presents a comprehensive checklist of Kubernetes best practices covering container image selection, registry authentication, namespace isolation, labeling, annotations, RBAC, pod security policies, network policies, secrets management, image scanning, CI/CD, canary releases, monitoring, service mesh, and admission controllers to help you build secure, stable, and scalable production clusters.
This guide walks through troubleshooting Kubernetes issues such as partially deleted resources, resetting etcd, apiserver start failures due to missing ServiceAccount certificates, SELinux permission errors, ServiceAccount key generation, etcd startup errors, host trust configuration, and resource limit pitfalls, providing concrete commands and scripts for each problem.
This talk presents NetEase's practical experience with Impala, covering its core architecture, new features in version 3.x, integration with Apache Iceberg, a custom management platform, profiling and statistics enhancements, as well as future plans involving Kubernetes, Alluxio caching and pre‑computation strategies.
k9s is a command‑line interface that streamlines Kubernetes cluster management by wrapping kubectl, offering real‑time resource tracking, metric monitoring, support for standard and custom resources, customizable UI, multi‑resource views, RBAC inspection, and easy navigation of related objects, with an official site for download.
This guide walks through setting up a cloud‑hosted Kubernetes control plane with a Raspberry Pi edge node, demonstrates the limitations of a native cluster under edge conditions, converts the cluster to OpenYurt using yurtctl, and evaluates operation commands and network‑disruption scenarios to showcase OpenYurt’s edge‑native capabilities.
This guide explains what Cerebro is, how to install it (including binary and Docker options), how to run it on Linux, macOS, and Windows, and how to use its UI to connect to an Elasticsearch node, view cluster overviews, manage shards, and execute DSL queries.
ZooKeeper, Apache’s distributed coordination service, offers a highly available in‑memory hierarchical file system with leader‑follower‑observer clustering and the ZAB protocol, guaranteeing sequential consistency, atomicity and a single view while supporting publish/subscribe, configuration management, distributed locks, master election and queueing for robust distributed applications.
This article explains how to build a real‑time data processing platform using Flink, covering the Lambda architecture, design approaches, SQL and custom‑Jar task definitions, UI drag‑and‑drop, cluster resource management on Yarn and Kubernetes, submission modes, scheduling, permission and metadata handling, logging, and monitoring with Prometheus and Grafana.
This article introduces a curated list of Kubernetes tools—including cluster deployment solutions, monitoring utilities, CLI helpers, and development aids—explaining how each simplifies container orchestration, enhances DevOps workflows, and empowers engineers to manage, observe, and extend their Kubernetes environments efficiently.
This article chronicles the four‑year evolution of Chehaoduo Group’s Kafka ecosystem—from its initial role as a simple data‑ingestion layer to becoming the core of the company’s large‑scale data pipeline—detailing cluster management, upgrade strategies, multi‑cluster deployment, AVRO schema handling, SDK development, and operational lessons learned.
This article explains the causes of ZooKeeper split‑brain situations, why odd‑numbered node deployments are preferred, how the quorum (majority) rule prevents split‑brain, and outlines practical methods such as quorum configuration, redundant communication, fencing, and pause‑before‑failover to handle and avoid the issue.
vivo built a high‑availability RabbitMQ middleware platform that combines an MQ‑Portal for request‑driven provisioning, an SDK that adds application‑level authentication, automatic cluster discovery, rate‑limiting, reset and blockage‑transfer capabilities, and a stateless MQ‑NameServer for name resolution and health‑based failover, enabling ten‑fold traffic growth without incidents.
Didi’s three‑year journey with Presto transformed it into the company’s primary ad‑hoc and Hive‑SQL acceleration engine, serving over 6 000 users, processing 2‑3 PB of HDFS data daily, and achieving major gains in stability, performance, cost, and usability through extensive architectural tweaks, resource isolation, connector extensions, and monitoring enhancements.
At Didi, Presto has been the default ad‑hoc and Hive‑SQL engine for over three years, serving 6,000 users, processing 2‑3 PB daily and 30‑35 trillion rows, with mixed and dedicated clusters, migration to PrestoSQL 340, extensive Hive compatibility, label‑based isolation, a native Druid connector, usability and stability enhancements, and JVM‑level performance optimizations, while planning further resource‑saving upgrades.
This article explains when Elasticsearch triggers a master election, describes each election stage—including active master and candidate selection, Bully algorithm comparison, and master node responsibilities—while providing code excerpts that illustrate the underlying implementation details.
This article provides a comprehensive analysis of Google’s Borg system, covering its design goals, user view, job and task model, resource allocation, scheduling algorithms, fault tolerance, scalability techniques, and operational metrics that shaped modern cloud‑native orchestration platforms.
Discover a curated list of powerful open-source Kubernetes management solutions—including K9s, Rancher, Dashboard, Kubectl, Kubeadm, Helm, KubeSpray, Kontena Lens, and WKSctl—detailing their core features, deployment options, and how they streamline cluster monitoring, configuration, and application lifecycle across cloud-native environments.
This article details 58.com’s extensive use of Elasticsearch for search, analytics, and log processing, covering cluster optimization challenges, typical issues like disk exhaustion and write slowdown, practical solutions, development standards, ELKB architecture, real‑time log and MySQL slow‑log applications, platform‑as‑a‑service construction, and future roadmap plans.
This guide explains how to troubleshoot Ceph near‑full OSD warnings, understand PG fault states, manage OSD and monitor statuses, perform cluster configuration changes without restarts, add or remove OSDs and monitors, adjust pool settings, and handle user permissions using detailed command examples.
This article describes JDHBase’s large‑scale KV storage, its HBase‑based replication principle, the multi‑active cluster architecture with Fox Manager, client routing, automatic failover, dynamic replication tuning, serial replication guarantees, and future directions for improving cross‑region disaster recovery.
Learn how to merge multiple kubeconfig files, resolve TLS user conflicts, and use the third‑party tools kubectx and kubens to quickly list, switch, and manage Kubernetes clusters and namespaces with simple command‑line shortcuts.
This article details AutoHome's use of Apache Kylin as its core OLAP engine, covering its development timeline, architecture, large‑scale cube deployment, performance optimizations, cluster upgrade experiences, and future directions such as real‑time OLAP and cloud‑native deployment.
This article details Autohome's use of Apache Kylin as its core OLAP engine, covering its architecture, large‑scale Cube deployment, real‑world business applications, a series of performance and operational optimizations, cluster upgrade experiences, and upcoming plans for real‑time OLAP and cloud‑native evolution.
Nacos 1.3.0 introduces an embedded relational database, unified cluster management, an upgraded Raft consistency protocol, and security fixes, providing simpler deployment, higher performance, and safer operations for both small‑scale and enterprise environments.
Nacos 1.3.0 introduces an embedded relational database, unified cluster management, an upgraded Raft consistency layer, security patches, Snowflake ID configuration, data migration guidance, new cluster addressing modes, and a set of Open‑API operations for Raft administration, all aimed at simplicity, performance, and high availability.
This article provides a comprehensive overview of Elasticsearch's internal architecture, explaining how data flows from memory buffers to Lucene segments, the role of refresh and translog for durability, segment merging strategies, shard routing, replica consistency, allocation controls, hot‑cold data separation, and cluster discovery settings.
Apache Zookeeper is an open‑source coordination service that provides reliable distributed synchronization, configuration management, and naming for big‑data components such as Hadoop, HBase, and Kafka, offering features like hierarchical znode structures, watches, master election, and distributed locks to maintain cluster health.
This article outlines a step‑by‑step procedure for safely taking an Elasticsearch node offline—checking master‑eligible settings, adjusting minimum_master_nodes, excluding the node from routing, waiting for shard relocation, stopping the service, and restoring the cluster routing—ensuring no data loss or service interruption.
This article explains ZooKeeper's role as a distributed coordination service, covering its architecture, znode types, watch mechanism, configuration management, distributed locking, queue handling, data replication, leader election, and synchronization processes for building reliable backend systems.
This guide walks you through the essential daily maintenance tasks for GaussDB T clusters, covering ETCD startup, cluster health checks, host resource monitoring, tablespace usage, abnormal wait events, log inspection, and common error troubleshooting with concrete commands and SQL examples.
This article describes JDHBase’s large‑scale KV storage deployment, its HBase‑based asynchronous replication mechanism, the multi‑active architecture with active‑standby clusters, client interaction via Fox Manager, automatic failover strategies, dynamic replication tuning, and serial replication techniques to ensure data consistency across data centers.
This article details Alibaba's massive Flink deployment, covering its historical background, the operational challenges of managing tens of thousands of nodes, the design of a comprehensive Flink management platform, and the automated solutions for fault handling, resource allocation, and performance testing in a large‑scale big‑data environment.
This article explains how Alibaba Cloud Container Service (ACK) designs a unit‑based, tiered management system, capacity planning model, global observability architecture, and pluggable components to reliably operate more than ten thousand diverse Kubernetes clusters during the massive Double‑11 shopping event.
Zhuanzhuan’s adoption of TiDB addressed sharding challenges and massive data storage needs, and the team shares six common issues encountered in large‑scale online deployments—including performance diagnosis, cluster management, log inconsistencies, slow‑SQL impact, optimizer limitations, and transaction conflicts—along with their standardized solutions for deployment, monitoring, alerting, and business rollout.
Ant Financial’s article details how its large‑scale Kubernetes management system—built on a meta‑cluster, end‑state operators, and a Kube‑on‑Kube design—ensures reliable creation, upgrade, and self‑healing of thousands of nodes, while providing gray‑scale changes, risk assessment, and fault‑tolerant automation.
This article explains how Ant Financial designs and operates a large‑scale, highly available Kubernetes management platform—detailing its architecture, core operators, desired‑state controllers, fault‑self‑healing mechanisms, risk mitigation, and practical Q&A for production environments.
Meituan‑Dianping describes its evolution from a custom Docker‑based scheduler (HULK1.0) to an open‑source Kubernetes‑based platform (HULK2.0), detailing architecture, resource‑management strategies, scheduler optimizations, Kubelet enhancements, and online‑cluster tuning that together enable stable, cost‑effective operation of a 100k+ node fleet.
This article explains how Ant Financial designs and operates a large‑scale, highly available Kubernetes management platform—using end‑state driven operators, custom CRDs, self‑healing mechanisms, and risk‑mitigation strategies—to reliably run thousands of nodes and dozens of business clusters in production.
This article details Meituan-Dianping's evolution from custom Docker‑based scaling to a Kubernetes‑driven, cloud‑native cluster management platform (HULK), describing its architecture, scheduler enhancements, Kubelet modifications, and resource‑optimization strategies for large‑scale operations.
This article shares a senior data‑platform engineer's hands‑on experience managing dozens of thousand‑node clusters, detailing nine common cluster problems and step‑by‑step solutions—including performance tuning, RPC fixes, HDFS cleanup, Hive metadata repair, Spark shuffle optimization, HBase region recovery, and Kafka bottleneck mitigation.
This article provides a comprehensive guide to Elasticsearch’s architecture, node roles, data organization, and the most important performance metrics—including search, indexing, memory, JVM garbage collection, host‑level system metrics, cluster health, and resource saturation—offering practical advice on monitoring and tuning the cluster for reliability and efficiency.
This article details Meituan‑Dianping's evolution from a custom Docker‑based cluster manager to the open‑source Kubernetes‑powered HULK2.0 platform, describing its architecture, operational practices, scheduler and Kubelet optimizations, and resource‑management techniques that enable massive, cost‑effective scaling.
Meituan‑Dianping’s evolution from virtualization to the HULK‑2.0 Kubernetes platform enables a 100,000‑instance, multi‑region cluster to achieve high elasticity and availability, using scheduler optimizations, local‑optimal placement, enhanced kubelet features, and fine‑grained resource management to maximize throughput during traffic spikes.
This article explains how Ant Financial designs a highly reliable, end‑state‑driven Kubernetes management platform that handles lifecycle operations, node self‑healing, and risk‑controlled changes for clusters with tens of thousands of nodes, using operators, custom resources, and a meta‑cluster architecture.
This article explains what large‑scale website operations entail, outlines the product lifecycle and the crucial role of operations engineers, lists essential technical skills and personal qualities, and discusses current challenges, future prospects, and key technical topics such as cluster management, monitoring, fault handling, and automation.
On April 27, Youzan Technology and the Elastic Chinese community hosted a “starry sky” technical salon where experts from Getui, Ant Financial, Haipai Ke and Youzan presented four talks on large‑cluster proxy management, multi‑tenant ES optimization, search‑platform engineering, and the evolution of Youzan’s log platform, followed by lively Q&A and resource sharing.
This article explains Zookeeper as an open‑source distributed coordination service, detailing its core functions such as cluster management, leader election, distributed locks, and naming service, along with its three deployment modes, state‑synchronization via the ZAB protocol, and its watcher‑based notification mechanism.
This article explores the definition, responsibilities, required skills, career challenges, and key technologies of large‑scale website operations, offering a comprehensive guide for aspiring and current operations engineers to understand and excel in this demanding field.
This article explains how to dramatically improve Elasticsearch throughput by using the bulk API, tuning bulk request sizes, configuring gateway settings, optimizing cluster state updates, managing caches, leveraging fielddata and doc values, and employing tools like Curator and the Profiler for efficient cluster operations.
Meltwater’s media‑monitoring platform runs a custom Elasticsearch 1.7.6 cluster of over 400 nodes on AWS, handling 200 TB of primary data and 3 million daily documents while serving thousands of complex queries per minute, achieved through careful shard design, master‑node configuration, extensive performance tuning, and automated provisioning.
This article describes how we identified underutilized CPU and memory resources in our company's servers, evaluated Kubernetes versus Apache Mesos, and built a non‑intrusive, Mesos‑based multi‑task scheduling system with dynamic resource reservation, monitoring, task isolation, and cluster‑wide observability, while addressing deployment challenges.
This article explains how we leveraged Apache Mesos to create a multi‑task scheduling system that maximizes idle CPU and memory on legacy CentOS machines without kernel upgrades, detailing architecture, deployment, monitoring, resource isolation, and remaining challenges.
Ant Financial tackles the challenge of managing dozens of Kubernetes clusters and over a hundred thousand worker nodes by employing a meta‑cluster with Kube‑on‑Kube and Node Operators, enabling automated lifecycle management, scaling, upgrades, and fault recovery for both master components and worker nodes.
The July 2018 Ceph Developer Monthly report from the UMCloud storage team summarizes the latest community contributions, including enhancements to object and block storage, new OPA integration for fine‑grained access control, crash‑dump management, dashboard user UI, and batch operations for ceph‑volume.
Scheduling algorithms are a crucial component of cluster resource management systems, determining where containerized tasks run to ensure resource needs, high availability, fault tolerance, and cost efficiency across individual containers, applications, and entire data centers, while also supporting Alibaba’s global scheduling challenge.
This article explains Spark's role in the DataMagic platform, outlines four practical steps to quickly master Spark, details key configuration and parallelism settings, shows how to modify Spark code, and provides operational tips for cluster management and job troubleshooting.
The April 2018 Ceph monthly report highlights LTTng tracing for RGW, SSL support for the beast frontend, MFA integration, a notrim option for rbd mapping, runtime lz4 and brotli compression, Zabbix PG metrics, asynchronous dashboard tasks, detailed operation tracking, osdmap pruning, and a new iostat manager plugin.
Facing the challenge of processing over 100 million daily parcels, Alibaba’s Cainiao designed a lightweight, time‑wheel‑based scheduling engine that decouples task storage from timing, leverages partitioned task chains, master‑driven node IDs, and cluster‑wide soft‑load balancing to achieve scalable, fault‑tolerant timer processing.
This article provides a comprehensive overview of IBM GPFS, covering its architecture, management components, networking models, cluster and storage design, as well as practical guidance on building‑block configurations for performance and capacity scaling in high‑performance computing environments.
This article shares practical insights on operating large‑scale Elasticsearch clusters for log analysis, covering use cases, essential tools, hardware choices, node role separation, shard management, hot‑cold data strategies, version upgrades, and key monitoring metrics to ensure stability and performance.
The team adopted Apache Ambari to streamline deployment, scaling, monitoring, and upgrade of their Hadoop‑centric big‑data platform, overcoming HA cluster takeover and custom Hive 2.1 integration through a three‑phase test, gray‑scale, and production rollout, thereby improving management efficiency and reducing O&M costs.
The article explains how rapid internet growth has expanded data centers, why traditional operations fall short, presents a simple utilization formula, shows Alibaba’s mixed offline‑online scheduling experiment that raised server usage from 10% to over 40%, and announces an open dataset for academic research.
This article reviews the latest Elasticsearch 5.x enhancements—including append‑only indexing, range fields, removal of the _all field, unified highlighter, keyword normalizer, multi‑word synonyms, field collapsing, cancellable searches, partitioned term aggregations, cluster allocation explain, Java REST client updates, cross‑cluster search, batched reduce phases—and previews the major features expected in Elasticsearch 6.0 such as sparse doc values, index sorting, sequence numbers, seamless rolling upgrades, type removal, index‑template inheritance, load‑aware shard routing, and X‑Pack extensions like SQL and machine learning.
The talk outlines how Docker and Kubernetes were adopted to streamline test environment provisioning, address challenges like environment inconsistency and resource scarcity, and enable automated, standardized, and scalable testing infrastructure through containerization, networking, storage, and cluster management techniques.
This article explains the architecture of Docker Swarm, the reasons for running it on Apache Mesos, the integration process—including resource offers, task scheduling, and container creation—and discusses current limitations and future improvement directions.
iQIYI’s Hadoop platform, built since 2010, now spans over a thousand nodes and 60 PB storage, detailing its architectural evolution, operational management practices, encountered challenges, and the custom Gear workflow system that streamlines job scheduling, dependencies, and alerts for large‑scale data processing.
This article explains the Mesos distributed system kernel, its resource‑allocation workflow, and how Qunar engineers applied and evolved Mesos, Marathon, and custom frameworks to achieve fine‑grained scheduling, high availability, service discovery, and multi‑tenant management in a large‑scale production environment.
This article shares practical experience on operating Zookeeper clusters, covering core concepts, deployment recommendations, configuration tuning, monitoring, migration strategies, and a list of common issues to avoid for reliable distributed coordination.
This article explains how Hulk’s private‑cloud platform customizes SaltStack for large‑scale command execution, detailing its three‑layer architecture, Redis‑based data flow, and a seven‑step workflow that achieves 99 % success while highlighting current limitations and future improvements.
This guide provides a comprehensive, command‑by‑command walkthrough for shutting down and starting Oracle RAC clusters, covering srvctl and crsctl usage, status checks, and essential options to help administrators manage database instances and cluster services reliably.
This article examines resource allocation philosophies—auction, budgeting, and preemption—and compares the architectures, data models, and APIs of major schedulers such as Borg, Omega, Mesos, Kubernetes, and Alibaba’s Zeus, while also exploring sharing strategies, task classifications, utilization metrics, and predictive techniques for efficient resource management.
This article walks through the shortcomings of manual code releases, designs a multi‑environment automated deployment workflow, details step‑by‑step implementation—including code fetching, configuration handling, logging, parallel execution, and rollback—while sharing practical scripts and common pitfalls for large‑scale clusters.
This guide explains Docker Swarm’s architecture, installation methods, and step‑by‑step procedures for creating a high‑availability Swarm cluster, including manager and node setup, TLS security, service discovery with Consul, and essential Docker commands for operating and testing the cluster.
This article provides a comprehensive overview of Elasticsearch 2.x, covering its distributed architecture, core concepts such as clusters, nodes, indices, shards and replicas, the ZenDiscovery master‑election process, scaling mechanisms, recovery, query features, and the underlying system components like Guice, Netty, and thread‑pool designs.
This article explains the Mesos distributed system kernel, its master‑slave architecture, fine‑grained resource scheduling, and how Qunar leverages Mesos and Marathon for log processing, Spark, Alluxio, and multi‑tenant services while addressing framework unification, HA, service discovery, and operational challenges.
This guide explains why the official Redis Cluster tools lack password support, outlines the security risks of an unauthenticated cluster, and introduces a custom management utility that adds password authentication, automates slot migration, and simplifies adding or removing nodes, complete with step‑by‑step testing procedures.
This article presents a detailed overview of Ctrip's Hermes message queue system, covering its architectural evolution from a simple Mongo‑based design to a broker‑centric, multi‑storage solution with meta‑server coordination, and discusses practical techniques for building high‑performance, scalable messaging infrastructure.
The article explains how Hadoop clusters suffer from resource contention among multiple users, why YARN alone often fails to prioritize workloads, and how Pepperdata provides deeper visibility and automatic adjustments that reduce low‑priority usage, cut node count, and lower cloud costs.
This article summarizes Yao Yu's "Scaling Redis at Twitter" talk, detailing why Twitter chose Redis, the massive memory and QPS requirements, custom data models, Hybrid List and BTree extensions, cluster management, and operational lessons for building a high‑performance caching service.
This guide explains the essential RHCS components—including CMAN, DLM, CCS, and FENCE—details how to start and stop the cluster, manage application services with clusvcadm, monitor cluster status using cman_tool, clustat, and ccs_tool, and maintain GFS2 file systems with dedicated utilities.
This guide explains the two scenarios for removing nodes from a Redis cluster—deleting a master node by first migrating its slots and then removing it, and deleting a slave node directly—along with the exact redis-trib.rb commands and verification steps to ensure successful removal.
This article provides a comprehensive overview of Google’s Borg system, detailing its purpose, user perspective, workload types, cluster and cell architecture, job and task management, scheduling algorithms, scalability techniques, and availability mechanisms for large‑scale distributed environments.
