How to Build a Secure High‑Availability Etcd Cluster on Linux

Environment Overview

Etcd is a distributed key‑value store written in Go that uses the Raft consensus algorithm to ensure data consistency across nodes. It is commonly used as a service registry and discovery mechanism in micro‑service architectures, offering simple deployment, data persistence, and SSL client authentication. Etcd joined the CNCF in late 2018 and graduated in November 2020.

Host Planning

This document assumes you replace all IP addresses with those of your own environment; the steps apply to other versions as well.

Three High‑Availability Cluster Forms

Static configuration – specify each node with --initial-cluster when starting etcd.

Etcd dynamic discovery – use an existing etcd cluster (e.g., discovery.etcd.io) to discover new members.

DNS dynamic discovery – obtain peer addresses via DNS queries.

1. Host Initialization

1.1 Configure /etc/hosts

cat <<EOF >> /etc/hosts
192.168.2.51 etcd1
192.168.2.52 etcd2
192.168.2.53 etcd3
EOF

1.2 Install cfssl Certificate Tools

wget https://github.com/cloudflare/cfssl/releases/download/v1.6.0/cfssl_1.6.0_linux_amd64 -O /usr/local/bin/cfssl
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.0/cfssljson_1.6.0_linux_amd64 -O /usr/local/bin/cfssljson
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.0/cfssl-certinfo_1.6.0_linux_amd64 -O /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl*

cfssljson converts JSON output from cfssl into PEM‑encoded certificates; cfssl‑certinfo displays detailed CSR or certificate information.

1.3 Create Working Directory

mkdir -p /etc/etcd/cert

/etc/etcd – etcd working directory

/etc/etcd/cert – certificate storage

1.4 Disable SELinux and firewalld

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
systemctl status firewalld.service
systemctl disable --now firewalld.service

2. Certificate Generation

2.1 Create Certificate Authority (CA)

Generate a default CA configuration file:

cfssl print-defaults config > /etc/etcd/cert/ca-config.json
cat <<EOF > /etc/etcd/cert/ca-config.json
{
  "signing": {
    "default": {"expiry": "87600h"},
    "profiles": {"etcd": {"expiry": "87600h","usages": ["signing","key encipherment","server auth","client auth"]}}
  }
}
EOF

Create a CSR for the CA:

cat <<EOF > /etc/etcd/cert/ca-csr.json
{
  "CN": "etcd",
  "key": {"algo": "rsa","size": 2048},
  "names": [{"C": "CN","ST": "BeiJing","L": "BeiJing","O": "etcd","OU": "system"}]
}
EOF

Generate the CA certificate:

cfssl gencert -initca /etc/etcd/cert/ca-csr.json | cfssljson -bare /etc/etcd/cert/etcd-ca

2.2 Issue Etcd Certificates

Create an etcd CSR:

cat <<EOF > /etc/etcd/cert/etcd-csr.json
{
  "CN": "etcd",
  "hosts": ["127.0.0.1","192.168.2.51","192.168.2.52","192.168.2.53"],
  "key": {"algo": "rsa","size": 2048},
  "names": [{"C": "CN","ST": "BeiJing","L": "BeiJing","O": "etcd","OU": "system"}]
}
EOF

Generate the etcd certificate using the CA:

cfssl gencert -ca=/etc/etcd/cert/etcd-ca.pem \
  -ca-key=/etc/etcd/cert/etcd-ca-key.pem \
  -config=/etc/etcd/cert/ca-config.json -profile=etcd \
  /etc/etcd/cert/etcd-csr.json | cfssljson -bare /etc/etcd/cert/etcd

Resulting files: etcd.pem (public cert), etcd-key.pem (private key), and etcd-ca.pem (CA cert).

2.3 Distribute Certificates

scp /etc/etcd/cert/{etcd-ca.pem,etcd.pem,etcd-key.pem} 192.168.2.52:/etc/etcd/cert/
scp /etc/etcd/cert/{etcd-ca.pem,etcd.pem,etcd-key.pem} 192.168.2.53:/etc/etcd/cert/

3. Deploy Etcd Cluster

3.1 Download Binary

wget -c https://github.com/etcd-io/etcd/releases/download/v3.5.5/etcd-v3.5.5-linux-amd64.tar.gz
tar -xf etcd-v3.5.5-linux-amd64.tar.gz
cp -p etcd-v3.5.5-linux-amd64/{etcd,etcdctl,etcdutl} /usr/local/bin/

etcd – server binary

etcdctl – client tool

etcdutl – data recovery utility

3.2 Create Etcd Configuration Files

Example for node etcd‑1 (repeat with appropriate IPs for etcd‑2 and etcd‑3):

# /etc/etcd/etcd.conf
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/etc/etcd/etcd-data"
ETCD_SNAPSHOT_COUNT="5000"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="500"
ETCD_LISTEN_PEER_URLS="https://192.168.2.51:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.2.51:2379,https://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.2.51:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.2.51:2380,etcd-2=https://192.168.2.52:2380,etcd-3=https://192.168.2.53:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-1"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.2.51:2379"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_TRUSTED_CA_FILE="/etc/etcd/cert/etcd-ca.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/cert/etcd-ca.pem"

3.3 Create Systemd Service File

# /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/etc/etcd/etcd.conf
ExecStart=/usr/local/bin/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

3.4 Start the Cluster

systemctl daemon-reload
systemctl start etcd.service
systemctl enable etcd.service
systemctl status etcd

3.5 Verify Cluster Health

etcdctl endpoint health --write-out=table \
  --endpoints=https://192.168.2.51:2379,https://192.168.2.52:2379,https://192.168.2.53:2379 \
  --cacert=/etc/etcd/cert/etcd-ca.pem \
  --cert=/etc/etcd/cert/etcd.pem \
  --key=/etc/etcd/cert/etcd-key.pem

The command returns a table indicating each endpoint is healthy.