Deploy NFS, ECK, Elasticsearch, Kibana & Filebeat on Kubernetes

1. Deploy NFS

1. Install NFS

# Install on all nodes
yum install -y nfs-utils

# Create shared directory on master
mkdir -pv /data/kubernetes

# Write /etc/exports
cat > /etc/exports <<'EOF'
/data/kubernetes *(rw,no_root_squash)
EOF

# Enable and start rpcbind and nfs on master
systemctl enable --now rpcbind nfs

2. Deploy NFS dynamic storage with Helm

a. Create namespace

# kubectl create ns nfs-sc-default

b. Download chart

# helm pull nfs-subdir-external-provisioner/nfs-subdir-external-provisioner

Inspect values.yaml after extracting.

# cat values.yaml | egrep -v '#|^$'
replicaCount: 1
strategyType: Recreate
image:
  repository: registry.cn-hangzhou.aliyuncs.com/haiweigit/nfs-subdir-external-provisioner
  tag: v4.0.2
  pullPolicy: IfNotPresent
nfs:
  server: 10.1.129.86
  path: /data/nfs-data
storageClass:
  create: true
  defaultClass: true
  name: nfs-sc-default
  allowVolumeExpansion: true
  reclaimPolicy: Delete

Install with the modified values.yaml:

helm install nfs-subdir-external-provisioner \
    /root/nfs/nfs-subdir-external-provisioner \
    -f values.yaml \
    -n nfs-sc-default

Check the created StorageClass:

# kubectl get sc
NAME               PROVISIONER                                   RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-sc-default (default)   cluster.local/nfs-subdir-external-provisioner   Delete          Immediate           true   38m

2. Introduction to ECK

2.1 What is ECK

To understand ECK, first know CRD, Controller, Operator.

What is CRD

Custom Resource Definitions allow extending Kubernetes with new resource types when built‑in types are insufficient.

What is Controller

A controller watches resource changes and reconciles the actual state to the desired state.

What is Operator

An operator uses CRDs to manage applications declaratively on Kubernetes.

What is ECK

Elastic Cloud on Kubernetes (ECK) is a Kubernetes Operator that simplifies deployment and management of the Elastic Stack (Elasticsearch, Kibana, APM, Beats, etc.).

2.2 ECK Features

Rapid deployment, management and monitoring of multiple clusters

Easy scaling of cluster size and storage

Rolling upgrades for configuration changes

TLS certificate protection

Hot‑warm‑cold architecture with zone awareness

2.3 Version support

For ECK 2.7 the supported component versions include:

Kubernetes 1.22‑1.26

OpenShift 4.8‑4.12

GKE, AKS, EKS

Helm 3.2.0+

Elasticsearch, Kibana, APM Server: 6.8+, 7.1+, 8+

Enterprise Search: 7.7+, 8+

Beats: 7.0+, 8+

Elastic Agent: 7.10+ (standalone), 7.14+ (Fleet), 8+

Elastic Maps Server: 7.11+, 8+

3. Cluster Deployment and Planning

3.1 Component versions

OS: CentOS 7.9 Kernel: 5.4.260-1.el7.elrepo.x86_64 Kubernetes: v1.23.17 Docker: 20.10.9 kube‑vip: 0.6.0 ECK: 2.7.0 ELK: 8.9.1

3.2 Environment preparation

3.2.1 System parameter tuning

Increase file descriptor limit

Set in /etc/profile

# vim /etc/profile
ulimit -n 65535
source /etc/profile

Configure /etc/security/limits.conf

# vim /etc/security/limits.conf
* soft nofile 65535
* hard nofile 65535

Verify

# ulimit -n
65535

Increase virtual memory map count

Temporary

# sysctl -w vm.max_map_count=262144
vm.max_map_count = 262144

Permanent

# cat >> /etc/sysctl.conf <<EOF
vm.max_map_count=262144
EOF
sysctl -p

Deploy ECK in the Kubernetes cluster

Quick start guide: https://www.elastic.co/guide/en/cloud-on-k8s/2.7/k8s-quickstart.html

Install CRDs

# wget https://download.elastic.co/downloads/eck/2.7.0/crds.yaml
# kubectl create -f crds.yaml

Install the operator and RBAC

# wget https://download.elastic.co/downloads/eck/2.7.0/operator.yaml
# kubectl apply -f operator.yaml

The operator runs in the elastic-system namespace; a dedicated namespace is recommended.

3.1 Deploy Elasticsearch cluster

Create resource manifest

cat <<EOF > elasticsearch.yaml
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
spec:
  version: 8.9.1
  http:
    tls:
      selfSignedCertificate:
        disabled: true
  nodeSets:
  - name: master
    count: 3
    config:
      node.store.allow_mmap: false
    podTemplate:
      spec:
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh','-c','sysctl -w vm.max_map_count=262144']
EOF

# kubectl apply -f elasticsearch.yaml

Monitor creation:

# kubectl get elasticsearch
NAME         HEALTH   NODES   VERSION   PHASE   AGE
elasticsearch   green    3       8.9.1     Ready   79s

Get service IP:

# kubectl get service elasticsearch-es-http
NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
elasticsearch-es-http ClusterIP   10.96.35.63    <none>        9200/TCP  3m25s

Retrieve default elastic user password:

# PASSWORD=$(kubectl get secret elasticsearch-es-elastic-user -o go-template='{{.data.elastic | base64decode}}')

3.2 Deploy Kibana instance

cat <<EOF > kibana.yaml
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
spec:
  version: 8.9.1
  count: 1
  elasticsearchRef:
    name: elasticsearch
  podTemplate:
    spec:
      containers:
      - name: kibana
        env:
        - name: I18N_LOCALE
          value: "zh-CN"
EOF

# kubectl apply -f kibana.yaml

Check status:

# kubectl get kibana

Expose Kibana service (ClusterIP by default) and optionally change to NodePort:

# kubectl edit service kibana-kb-http   # set type: NodePort
# kubectl port-forward --address 0.0.0.0 service/kibana-kb-http 5601

3.3 Deploy Filebeat

ConfigMap with filebeat.yml:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: default
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
    processors:
    - add_cloud_metadata:
    - add_host_metadata:
    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}
    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}

DaemonSet definition (truncated for brevity):

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: default
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:7.15.2
        args: ["-c","/etc/filebeat.yml","-e"]
        env:
        - name: ELASTICSEARCH_HOST
          value: elasticsearch-es-http
        - name: ELASTICSEARCH_PORT
          value: "9200"
        - name: ELASTICSEARCH_USERNAME
          value: elastic
        - name: ELASTICSEARCH_PASSWORD
          value: 0hha56I2uWDtu28Q4l4j4eW9
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
      volumes:
      - name: config
        configMap:
          name: filebeat-config
          defaultMode: 0640
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate

Apply RBAC objects (ClusterRole, Role, RoleBinding, ServiceAccount) as shown in the source.

# kubectl apply -f filebeat-kubernetes.yaml

For more details see the original article: https://www.cnblogs.com/boradviews/p/18240997