This article explains the distributed key‑value store etcd, its origin, core characteristics such as simplicity, security, speed and Raft‑based reliability, and details eight practical scenarios—including service discovery, pub/sub, load balancing, distributed locks and leader election—while also comparing it with Zookeeper.
Configuration ManagementRaftdistributed key-value store
This article explains service governance concepts, the challenges of managing thousands of micro‑services, introduces etcd and its Raft‑based consistency model, details BoltDB storage internals, and describes Baidu's large‑scale Tianlu platform with its high‑availability, performance, scalability, and operational metrics.
This guide explains why etcd is critical for Kubernetes, walks through creating snapshots with etcdctl, automating backups via scripts and cron, and details step‑by‑step procedures for restoring a failed etcd cluster, including stopping services, cleaning data directories, and restarting components to recover the whole cluster.
This article introduces service governance fundamentals, explains how etcd’s Raft‑based consensus and BoltDB storage work, compares etcd with ZooKeeper and Consul, and describes Baidu’s large‑scale, high‑availability, high‑performance service‑governance platform built on these technologies.
This article explores service governance fundamentals, examines why etcd’s Raft‑based consensus and BoltDB storage make it ideal for large‑scale systems, compares it with ZooKeeper and Consul, and shares Baidu’s practical architecture, performance tricks, and operational metrics for high‑availability, high‑performance service management.
This article examines Ant Group's strategies for achieving high availability of the etcd key‑value store in a massive 10,000‑node Kubernetes cluster, detailing challenges, performance metrics, filesystem upgrades, tuning parameters, operational platform insights, and future directions for distributed etcd deployments.
This guide explains how to prepare large‑scale Kubernetes clusters on public clouds by expanding node quotas, tuning kernel parameters, configuring high‑availability etcd, adjusting kube‑apiserver limits, and applying pod‑level resource and affinity best practices.
This article details the design, deployment, configuration, operation, monitoring, and backup of a PostgreSQL high‑availability cluster built on Patroni, Etcd, and LVS at 360, covering hardware layout, software versions, installation steps, parameter tuning, fail‑over testing, and future outlook.
This guide outlines essential steps for scaling large Kubernetes clusters on public clouds, covering node quota adjustments, kernel parameter tuning, etcd high‑availability setup, API server and pod configurations, and best‑practice recommendations to ensure stable performance as node counts grow.
The article describes how Ant Group operates a world‑largest Kubernetes deployment of over 10,000 nodes, details the performance challenges of the etcd key‑value store at such scale, and outlines a comprehensive set of hardware upgrades, configuration tuning, monitoring, data‑splitting, and future distributed‑etcd strategies to achieve robust high‑availability.
This article introduces Apache APISIX, examines the challenges of using it as a data‑plane in a service‑mesh architecture, presents the apisix‑mesh‑agent as an intermediary solution, and outlines the advantages, design, and future roadmap of a APISIX‑based cloud‑native service mesh.
This article provides a concise, question‑driven overview of Kubernetes, explaining the roles of master and worker nodes, pod networking, scheduling, storage with etcd, service exposure, scaling mechanisms, and how the various controllers collaborate to manage a cloud‑native cluster.
This guide walks you through setting up a three‑node etcd cluster on CentOS 7 using static configuration and self‑signed TLS certificates, covering member addition and removal, data backup via snapshots, and full cluster restoration from those snapshots.
This guide walks through essential Kubernetes operations such as isolating and recovering nodes, expanding clusters with new nodes, dynamically scaling Pods, managing Labels, scheduling Pods to specific Nodes, performing rolling updates, and configuring high‑availability for etcd and Master components, all with concrete command‑line examples and YAML snippets.
The article analyzes Bilibili's recent half‑hour service disruption, explores technical rumors such as an etcd crash, examines Kubernetes‑based cloud‑native infrastructure, reviews similar historic outages, and offers expert recommendations for improving high‑availability and disaster‑recovery in large‑scale internet services.
This guide walks through upgrading a HA Kubernetes cluster—from updating kubeadm, kubelet, and kubectl on master and worker nodes—to safely migrating an etcd cluster, covering version compatibility, backup procedures, and step‑by‑step commands to minimize service interruption.
This article explains common database logging mechanisms such as MySQL redo logs and binlogs, compares them with Redis AOF and etcd's Raft‑based WAL, and provides an in‑depth analysis of etcd's WAL source code, including key structures, creation process, record types, encoding, and file pipeline management.
This article explains how etcd 3.x switched its external API to gRPC, the challenges of using its gRPC‑gateway for HTTP requests in Apache APISIX, the default message size limit causing sync failures, and the certificate configuration pitfalls that were fixed through a PR merged in v3.5.0.
This article explains the core mechanisms of the Raft consensus algorithm—including leader election, log replication, safety guarantees, message structures, state transitions, and key Go implementations in etcd-raft—providing code examples and detailed analysis of functions such as becomeLeader, tickElection, and appendEntry.
This article explains etcd as a highly available distributed key‑value store, outlines its simple, secure, fast, and reliable characteristics, describes typical scenarios such as service discovery and distributed locking, and then provides a comprehensive overview of Kubernetes architecture, components, deployment methods, security, networking, storage, and operational best practices.
This guide explains why etcd backup is critical for Kubernetes disaster recovery, walks through snapshot creation, distribution, scheduled cron jobs, and provides a step‑by‑step procedure to restore the cluster on all nodes, ensuring services resume correctly.
This guide presents a concise, step‑by‑step roadmap for mastering Go, covering beginner fundamentals, essential packages, and advanced cloud‑native projects like Etcd, Docker, Kubernetes, and Istio, while offering practical study tips and career insights.
Accidental namespace deletions in Kubernetes can cause massive data loss, but by using etcd snapshots, resource‑level backup tools like Velero, PX‑Backup, and Kasten, and configuring scheduled backups, hooks, and PVC migration, you can protect clusters, streamline recovery, and avoid painful manual redeployments.
This article explains the concept of distributed locks, outlines common application scenarios, and provides detailed Java implementations using Redis (including Redisson and RedLock), Etcd, and Zookeeper, complete with code examples and a comparative summary of their advantages and drawbacks.
This guide explains the fundamentals of etcd, its Raft‑based architecture, cluster planning, secure certificate generation, installation steps, service configuration, and verification commands to deploy a reliable, SSL‑protected etcd cluster for service discovery and configuration management.
This article explains the fundamentals of distributed locks, compares lock implementations based on asynchronous replication and Paxos protocols, and provides practical Redis and etcd/Zookeeper examples—including exclusive and shared lock mechanisms, code snippets, and usage considerations for reliability and safety.
This guide walks through troubleshooting Kubernetes issues such as partially deleted resources, resetting etcd, apiserver start failures due to missing ServiceAccount certificates, SELinux permission errors, ServiceAccount key generation, etcd startup errors, host trust configuration, and resource limit pitfalls, providing concrete commands and scripts for each problem.
This article presents JD's self‑developed Remote Shuffle Service for Spark, detailing its architecture, goals, implementation details, performance benchmarks, and real‑world production case studies that demonstrate its impact on shuffle efficiency and system stability in large‑scale data processing.
This article provides a concise yet comprehensive overview of etcd, covering its background, core features, common application scenarios such as key‑value storage, service discovery, messaging and distributed locking, and detailed step‑by‑step installation methods for various platforms.
This article walks through designing and implementing a Go log‑collection agent that uses etcd for configuration storage, context for timeout and metadata handling, and Kafka for message consumption, complete with code examples, setup instructions, and a rate‑limiting utility.
This guide walks through troubleshooting common Kubernetes issues such as deleting stuck RCs, Deployments, and Services, resetting etcd after failures, fixing apiserver start errors caused by missing ServiceAccount certificates, handling SELinux permission denials, configuring host trust, and force‑deleting problematic Pods or Namespaces.
This tutorial walks through configuring server hosts, generating TLS certificates with cfssl for etcd and Kubernetes components, deploying an etcd cluster and Kubernetes master services (apiserver, controller‑manager, scheduler) via systemd, setting up kubelet and kube‑proxy on worker nodes, installing Docker, applying Flannel CNI, and adding additional worker nodes to the cluster.
This guide walks through downloading the etcd binary, preparing TLS certificates and configuration files, setting up systemd service management, and using Ansible playbooks to deploy and verify an etcd cluster across multiple nodes.
This article provides a comprehensive overview of etcd, covering its core concepts, common application scenarios, multiple installation methods, v3 architecture, command‑line client usage, and secure TLS deployment for cloud‑native distributed systems.
This tutorial walks through preparing the hostnames and /etc/hosts entries, generating a root CA and node certificates with cfssl, distributing those certificates, installing and configuring kubectl, creating admin credentials and kubeconfig, and finally deploying an etcd cluster with systemd units on a multi‑node Kubernetes environment.
The article explains the background, architecture, data distribution, processing flow, replication mechanisms, and management features of Pika's native distributed cluster, detailing how Etcd, LVS, and RocksDB are used to achieve scalable, persistent Redis-compatible storage with table isolation and flexible slot replication.
This guide walks through installing etcd, launching a static three‑node pseudo‑cluster on a single host, explains key command‑line flags, and shows how to deploy, configure, and scale the etcd cluster in Kubernetes using a corrected StatefulSet manifest.
This article explains what etcd is, outlines common failure points, and provides step‑by‑step instructions for collecting etcd metrics via curl, configuring Prometheus scraping, creating alerts, and using Sysdig Monitor to observe key health indicators in a Kubernetes environment.
This article explains etcd’s Raft‑based consensus, why odd‑numbered nodes are recommended, details the leader election process with log excerpts, discusses split‑brain and consistency guarantees, and provides step‑by‑step instructions for generating certificates, deploying an etcd cluster, and using etcdctl commands.
Facing frequent service outages as their microservice count grew, MasterTeach migrated from Eureka to Alibaba Nacos, designing a high‑availability Nacos‑Eureka sync solution with consistent‑hash sharding, Zookeeper/Etcd coordination, automated DevOps integration, and extensive fault‑tolerance testing to ensure stable operation of over 660 services.
This guide explains what etcd is, how to prepare a Kubernetes environment, start single‑node or multi‑node clusters, secure communication with TLS, replace failed members, perform built‑in and volume backups, scale the cluster, recover from snapshots, upgrade to etcd 3, and avoid known client‑balancer bugs.
This article details how Tencent Cloud’s TKE team identified, analyzed, reproduced, and resolved multiple etcd stability and performance issues—including data inconsistency, memory leaks, mvcc deadlocks, and WAL crashes—while sharing the lessons learned and the optimizations applied to support million‑node Kubernetes deployments.
The article introduces the JdHotkey framework, a lightweight, real‑time hotkey detection solution for Redis and other services, detailing the risks of hot keys, prior mitigation methods, core design components, operational workflow, and performance benchmarks across various hardware configurations.
The article examines the challenges of managing many services without fixed ports, explains the concepts of service registration and discovery, and compares popular tools such as Zookeeper, etcd, Consul, Registrator and confd, highlighting their strengths, weaknesses, and suitable use‑cases in modern distributed systems.
This article explains what etcd is, describes a master‑standby high‑availability scenario, and provides a step‑by‑step Java implementation with jetcd, including Maven dependency, client initialization, lease and lock APIs, and complete test cases that demonstrate automatic failover.
Youzan redesigned its ID generator to run independent March instances per room, sharing a single etcd cluster but allocating distinct, non‑overlapping ID ranges via boundary, lower, and upper parameters, enabling room‑level high availability, horizontal scaling, and zero‑downtime migration with automated tooling.
This comprehensive guide explains what etcd is, its Raft‑based architecture, key terminology, read/write flow, leader election, common scenarios such as service discovery and distributed locking, and provides step‑by‑step instructions for single‑node and multi‑node deployments with practical command examples.
This tutorial walks through building a production‑grade Kubernetes cluster on CentOS by provisioning five virtual machines, configuring networking and DNS, generating TLS certificates with cfssl, installing Docker, setting up a private Harbor registry, and deploying core Kubernetes components such as etcd, kube‑apiserver, controller‑manager, scheduler, kubelet and kube‑proxy with full supervision and health checks.
This tutorial walks you through preparing the environment, configuring cluster components, installing required packages, setting up Python 3.7, running pre‑install checks, executing the GaussDB installation script, handling common errors, and finally uninstalling the cluster on a CentOS 7.5 system.
This article chronicles the design, evolution, and performance evaluation of Bond, a distributed‑lock SDK used internally at Youzan, covering its multi‑phase roadmap, storage choices, timeout‑retry strategies, lock‑TTL recommendations, and practical pitfalls for reliable operations.
This article details the 360 team’s decision to adopt Apache APISIX for their gateway layer, describes the production architecture, containerized deployment steps, plugin development workflow, and operational metrics, providing a practical guide for building and managing a high‑performance API gateway.
Tangram is a home‑grown configuration center that replaces Spring Config Server by leveraging etcd’s high‑availability KV store, watch‑based dynamic updates, and multi‑language support to provide reliable, secure, and real‑time configuration management for microservice architectures.
This article explains the inner workings of etcd, identifies performance bottlenecks across Raft, storage, and network layers, and provides concrete server‑side hardware and software tuning, a new O(1) freelist algorithm, plus client‑side usage guidelines to run a stable, high‑throughput etcd cluster.
This article compares Zookeeper and etcd, detailing their origins, architecture, APIs, strengths, and weaknesses, to help you decide which distributed coordination system best matches your project's requirements and programming language preferences.
This article traces etcd’s evolution from its CoreOS origins, explains its Raft‑based distributed architecture, details its API groups, versioning and watch mechanisms, and showcases typical scenarios such as metadata storage, service discovery, leader election, and distributed coordination in cloud‑native environments.
This article examines Google's high‑availability distributed cron design, distills its core requirements and algorithms, and then presents a streamlined implementation for a startup using etcd and Raft, followed by a thoughtful discussion on whether early‑stage companies should adopt a middle‑platform strategy.
This guide introduces APISIX, a high‑performance, extensible microservice API gateway built on Nginx and etcd, outlines its key features, and provides step‑by‑step instructions for installing OpenResty, etcd, launching APISIX, and testing rate‑limiting capabilities.
This article details how Alibaba and Ant Financial tackled the performance and stability challenges of running Kubernetes at massive scale, describing their enhancements to etcd, API Server, scheduler, controller failover, load balancing, and other components that enabled a 10k‑node cluster to support the 2019 Tmall 618 promotion.
This article details Alibaba's experience deploying Kubernetes at massive scale, describing the performance bottlenecks encountered in etcd, API server, controller, and scheduler components, and presenting the concrete engineering improvements—such as storage sharding, lease‑based heartbeats, load‑balancing, watch bookmarks, and hot‑standby controllers—that enabled stable operation of clusters with tens of thousands of nodes.
This article reviews the evolution of etcd’s read‑write mechanisms, explains the “Fully Concurrent Read” feature introduced in etcd 3.4, and presents experimental results showing how it dramatically reduces expensive read latency and improves overall throughput in Kubernetes clusters.
This guide explains ETCD’s high‑availability architecture and provides detailed step‑by‑step recovery procedures for single‑node failures, majority‑node outages, and database‑space‑exceeded errors, including status checks, member removal and addition, snapshot restoration, compaction, defragmentation, and alarm clearing.
The article analyzes a slowdown issue in the 58 Cloud Platform caused by an overloaded API Server and uneven Etcd load, explains the root causes—including load‑balancing failure and missing namespace segmentation—and presents concrete remediation steps such as DNS round‑robin, namespace partitioning, Etcd client upgrade, and cache‑enabled queries.
This article explains why Alibaba needed to boost etcd’s storage capacity, analyzes the bottleneck caused by BoltDB’s freelist spill operation, presents a hash‑based redesign that reduces allocation and release from O(n) to O(1), and shows benchmark results confirming dramatic latency improvements for large‑scale deployments.
Chinese cloud‑native pioneer Li Xiang, the first Chinese member of the CNCF Technical Oversight Committee, discusses his open‑source contributions, Alibaba’s large‑scale cluster management, the evolution of cloud‑native technologies, and offers advice for young engineers on mastering code, design, and emerging trends.
The paper presents a highly scalable, low‑overhead Host‑based Intrusion Detection System architecture designed for hundreds of thousands of servers, emphasizing cluster high‑availability, strong consistency via a CP‑oriented etcd backend, Go‑based agents with efficient resource management, modular sandboxing, and robust process monitoring to ensure reliable, secure operation at massive scale.
This article details the requirements analysis, design choices, architecture, code implementation, monitoring, and special considerations of ZZLock, a high‑performance distributed lock solution built on Etcd that ensures atomicity, consistency, and fault‑tolerance for backend services.
The article outlines etcd's evolution from its 2013 inception to becoming a CNCF‑graduated project, highlights its core raft implementation and widespread adoption in Kubernetes and major distributed databases, and details Alibaba's extensive use and future development plans for the system.
This article explains how etcd stores and retrieves multiple revisions of a key using a B‑tree index, details the read and write paths, describes index restoration on startup, and shows how the watch subsystem delivers change events to clients, all illustrated with Go code snippets.
This article explains the inner workings of etcd, covering its Raft consensus implementation, node election and state management, as well as the persistent storage layer built on BoltDB, and includes detailed code excerpts to illustrate each component.
The article summarizes the 58 Group technical salon where experts compared Meituan‑Dianping's Cellar and 58's WTable distributed KV storage systems, detailing their architectures, improvements, scalability, high‑availability mechanisms, and operational considerations, and concludes with a comparative analysis and preview of the next session.
This article examines the shortcomings of a legacy log‑collection setup, introduces a streamlined real‑time architecture that eliminates per‑machine Hadoop clients, centralizes configuration via Etcd with a web UI, and details the Go implementation—including configuration design, Etcd watching, Kafka integration, tail management, and per‑second rate limiting—complete with code snippets and performance graphs.
This article describes the shortcomings of a legacy log‑collection architecture, proposes a streamlined real‑time design that centralises configuration in Etcd, uses a single Go‑based logagent to tail files, applies per‑service rate limiting, and forwards logs to Kafka for downstream processing.
This tutorial walks you through the complete process of manually installing a Kubernetes 1.9.9 master‑worker cluster on CentOS 7, covering environment preparation, binary downloads, configuration of etcd and Kubernetes components, creation of systemd services, and verification of the deployment.
The article explains that Eureka 2.0's open‑source development has been halted, advises developers to assess whether they are using the discontinued 2.x version, and outlines safe migration paths to alternatives like Consul, ZooKeeper, or Etcd while emphasizing cautious decision‑making.
This article analyzes the challenges of mocking Dubbo services in large‑scale Java projects, explores multiple design options—including generic service registration, Javassist proxies, and custom reference implementations—compares their pros and cons, and presents the final architecture that enables dynamic registration, method‑level mocking, and transparent fallback to real services.
This article compares three popular service discovery solutions—Zookeeper, etcd, and Consul—detailing their architectures, features, integration methods, strengths, and weaknesses, and concludes with a recommendation for using Consul in multi‑data‑center environments while noting its real‑time notification limitations.
This article explains Pandora.js's Service mechanism—a high-level process orchestration feature for Alibaba's Node.js applications—detailing its role in middleware management, standardized service lifecycles, isolated tasks, and an Etcd‑based service discovery example.
This article explains Pandora.js's Service mechanism—a high‑level process orchestration feature for Node.js applications—covering its problem scope, core principles, capabilities, implementation of an RPC provider with etcd registration, and a simple consumer example, while demonstrating start/stop lifecycle management.
The article offers a clear, beginner‑friendly overview of Kubernetes architecture—including master‑slave roles, key components like etcd and kube‑apiserver, the registration‑watch workflow, and custom controllers such as lb‑controller and log‑controller—while inviting readers to share feedback and explore deeper implementation details.
This article explains how Docker containers are managed within Kubernetes clusters using the Flannel component, detailing its interaction with etcd to allocate unique subnet ranges for each minion node, configuring flanneld, and troubleshooting common networking issues.
This article analyzes Etcd's distributed architecture, Raft‑based consistency, storage implementation, watch and lease mechanisms, differences between v2 and v3, and compares it with Zookeeper and Consul, providing practical usage tips and surrounding tooling for developers of distributed systems.
This article explains how to design and implement a reliable etcd disaster‑recovery system for a Kubernetes‑based platform, covering backup scheduling, data synchronization across sites, code structure, and the advantages and drawbacks of the approach.
This tutorial walks through building a complete Kubernetes cluster on RedHat 7 using yum‑based packages, covering etcd installation, master and minion setup, container‑to‑container networking with quagga, and optional kube‑ui deployment, while highlighting common pitfalls and providing reference resources.
This article introduces etcd, a highly‑available key‑value store based on the Raft algorithm, and explores its classic use cases such as service discovery, messaging, load balancing, distributed coordination, locks, queues, monitoring, leader election, and compares it with ZooKeeper, highlighting its simplicity, security, and cloud‑native advantages.
This article outlines Meipai’s evolution into a billion‑user short‑video platform, detailing its high‑availability, scalable architecture, service discovery via etcd, data storage challenges, CDN and cloud‑storage redundancy, fault‑tolerance mechanisms, and future directions such as H.265 adoption and P2P‑CDN hybrid delivery.
This article walks through installing and configuring a Kubernetes 0.6.2 cluster on CentOS 7, covering core concepts, environment setup, master and minion configuration, deployment of pods, services, replication controllers, and practical testing of load balancing and scaling.
etcd, a highly‑available key‑value store built on the Raft algorithm, provides simple HTTP/JSON APIs for secure, fast, and reliable shared configuration and service discovery, enabling use cases such as service registration, load balancing, distributed locks, queues, leader election, and real‑time cluster monitoring.