How to Build a Secure High‑Availability Etcd Cluster on Linux

Etcd Overview

Etcd is a distributed key‑value store written in Go that uses the Raft consensus algorithm to ensure data consistency across a cluster. It is commonly used as a service registry and discovery component in micro‑service architectures, offering simple deployment, data persistence, and SSL client authentication. Etcd graduated to CNCF in November 2020 and is a core cloud‑native storage component.

Environment Setup

Plan three hosts for the cluster (e.g., 192.168.2.51, 192.168.2.52, 192.168.2.53) and ensure the IP addresses are reachable.

All IP addresses in the following steps should be replaced with those of your own environment.

High‑Availability Cluster Forms

Static configuration – specify each node with --initial-cluster at startup.

Etcd dynamic discovery – use an existing etcd cluster as a discovery service (e.g., discovery.etcd.io).

DNS dynamic discovery – resolve peer addresses via DNS queries.

1. Host Initialization

1.1 Configure /etc/hosts

$ cat <<EOF >> /etc/hosts
192.168.2.51 etcd1
192.168.2.52 etcd2
192.168.2.53 etcd3
EOF

1.2 Install cfssl Certificate Tools

$ wget https://github.com/cloudflare/cfssl/releases/download/v1.6.0/cfssl_1.6.0_linux_amd64 -O /usr/local/bin/cfssl
$ wget https://github.com/cloudflare/cfssl/releases/download/v1.6.0/cfssljson_1.6.0_linux_amd64 -O /usr/local/bin/cfssljson
$ wget https://github.com/cloudflare/cfssl/releases/download/v1.6.0/cfssl-certinfo_1.6.0_linux_amd64 -O /usr/local/bin/cfssl-certinfo
$ chmod +x /usr/local/bin/cfssl*

cfssljson converts JSON output from cfssl into PEM certificates; cfssl‑certinfo displays detailed CSR or certificate information.

1.3 Create Working Directory

$ mkdir -p /etc/etcd/cert

/etc/etcd – etcd working directory.

/etc/etcd/cert – directory for certificates.

1.4 Disable SELinux and firewalld

$ sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
$ systemctl status firewalld.service
$ systemctl disable --now firewalld.service

2. Certificate Generation

2.1 Create Certificate Authority (CA)

Generate a default CA configuration file:

$ cfssl print-defaults config > /etc/etcd/cert/ca-config.json

Create ca-config.json with the following content:

{
  "signing": {
    "default": {"expiry": "87600h"},
    "profiles": {
      "etcd": {
        "expiry": "87600h",
        "usages": ["signing","key encipherment","server auth","client auth"]
      }
    }
  }
}

Explanation:

default.expiry : default certificate validity period (hours).

profiles.etcd : settings used when issuing etcd certificates.

signing : enables the certificate to sign other certificates.

key encipherment : allows key encryption.

server auth and client auth : enable mutual TLS authentication.

2.2 Generate CSR Files

$ cfssl print-defaults csr > /etc/etcd/cert/ca-csr.json
$ cat <<EOF > /etc/etcd/cert/ca-csr.json
{
  "CN": "etcd",
  "key": {"algo": "rsa", "size": 2048},
  "names": [{"C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "etcd", "OU": "system"}]
}
EOF

Fields such as CN , C , ST , L , O define the certificate subject.

2.3 Issue CA Certificate

$ cfssl gencert -initca /etc/etcd/cert/ca-csr.json | cfssljson -bare /etc/etcd/cert/etcd-ca

The command creates etcd-ca.pem, etcd-ca-key.pem, and etcd-ca.csr in /etc/etcd/cert/.

2.4 Issue Etcd Node Certificates

$ cfssl print-defaults csr > /etc/etcd/cert/etcd-csr.json
$ cat <<EOF > /etc/etcd/cert/etcd-csr.json
{
  "CN": "etcd",
  "hosts": ["127.0.0.1","192.168.2.51","192.168.2.52","192.168.2.53"],
  "key": {"algo": "rsa", "size": 2048},
  "names": [{"C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "etcd", "OU": "system"}]
}
EOF
$ cfssl gencert -ca=/etc/etcd/cert/etcd-ca.pem \
    -ca-key=/etc/etcd/cert/etcd-ca-key.pem \
    -config=/etc/etcd/cert/ca-config.json -profile=etcd \
    /etc/etcd/cert/etcd-csr.json | cfssljson -bare /etc/etcd/cert/etcd

This produces etcd.pem, etcd-key.pem, and etcd.csr.

2.5 Distribute Certificates

$ scp /etc/etcd/cert/{etcd-ca.pem,etcd.pem,etcd-key.pem} 192.168.2.52:/etc/etcd/cert/
$ scp /etc/etcd/cert/{etcd-ca.pem,etcd.pem,etcd-key.pem} 192.168.2.53:/etc/etcd/cert/

3. Deploy Etcd Cluster

3.1 Download Binary

$ wget -c https://github.com/etcd-io/etcd/releases/download/v3.5.5/etcd-v3.5.5-linux-amd64.tar.gz
$ tar -xf etcd-v3.5.5-linux-amd64.tar.gz
$ cp -p etcd-v3.5.5-linux-amd64/{etcd,etcdctl,etcdutl} /usr/local/bin/
$ etcd -version

Etcd provides three binaries: etcd (server), etcdctl (client), and etcdutl (data recovery).

3.2 Create Etcd Configuration Files

Example /etc/etcd/etcd.conf for node etcd-1:

# Member
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/etc/etcd/etcd-data"
ETCD_SNAPSHOT_COUNT="5000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="500"
ETCD_LISTEN_PEER_URLS="https://192.168.2.51:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.2.51:2379,https://127.0.0.1:2379"
# Clustering
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.2.51:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.2.51:2380,etcd-2=https://192.168.2.52:2380,etcd-3=https://192.168.2.53:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-1"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.2.51:2379"
# Security
ETCD_CLIENT_CERT_AUTH="true"
ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_TRUSTED_CA_FILE="/etc/etcd/cert/etcd-ca.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/cert/etcd-ca.pem"

Create similar files for etcd-2 and etcd-3 with the appropriate IP addresses.

3.3 Create Systemd Service

# /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/etc/etcd/etcd.conf
ExecStart=/usr/local/bin/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

3.4 Start the Cluster

$ systemctl daemon-reload
$ systemctl start etcd.service
$ systemctl enable etcd.service
$ systemctl status etcd

3.5 Verify Cluster Health

$ etcdctl endpoint health --write-out=table \
  --endpoints=https://192.168.2.51:2379,https://192.168.2.52:2379,https://192.168.2.53:2379 \
  --cacert=/etc/etcd/cert/etcd-ca.pem \
  --cert=/etc/etcd/cert/etcd.pem \
  --key=/etc/etcd/cert/etcd-key.pem

The command returns a table indicating the health status of each endpoint.