Self-Hosted Kubernetes Daily Inspection Checklist: Core Components & Resources

Introduction: Why Self-Hosted Kubernetes Needs Daily Inspection

Self-hosted Kubernetes clusters lack cloud vendor automatic monitoring and fault tolerance; cluster stability depends entirely on operational control. Daily inspection acts like a "health check" — regularly checking core component status and resource usage to detect potential risks early (component abnormal restarts, resource exhaustion, network disconnections) and prevent small issues from becoming production failures. This article provides a practical inspection checklist covering two core dimensions: "Component Status" and "Resource Usage," including specific commands, judgment criteria, and anomaly handling suggestions.

1. Core Component Status Inspection Checklist

Core components are the cluster's "skeleton"; focus on control plane, nodes, network, and storage modules to ensure each component runs normally and communicates smoothly.

1.1 Control Plane Components (Cluster "Brain")

The control plane consists of kube-apiserver, etcd, kube-controller-manager, and kube-scheduler; any component failure affects cluster scheduling and management.

kube-apiserver

Check Target: No frequent restarts, authentication/etcd connection normal

Check Commands:

# 1. Pod status
kubectl get pods -n kube-system -l component=kube-apiserver
# 2. Logs for apiserver errors
journalctl -u kubelet -g "apiserver" --no-pager

Normal Standard: Pod status Running, restart count ≤0 in 7 days; logs without ERROR level etcd timeout / auth failed Anomaly Handling:

Frequent restarts: check logs to locate cause (e.g., certificate expiry, etcd connection anomaly)

Inaccessible: check --bind-address parameter and port (6443) openness

etcd (database)

Check Target: Cluster members normal, data undamaged

Check Commands:

# 1. Endpoint health
kubectl -n kube-system exec -it etcd-<node-name> -- etcdctl endpoint health --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key
# 2. Member list
kubectl -n kube-system exec -it etcd-<node-name> -- etcdctl member list

Normal Standard: All nodes health ok; member status started, no unhealthy nodes

Anomaly Handling:

Single point failure: if multi-node etcd, manually promote standby node

Data corruption: restore from backup (requires pre-configured etcd snapshots)

controller-manager

Check Target: Controllers running normally (e.g., node controller)

Check Commands:

# 1. Pod status
kubectl get pods -n kube-system -l component=kube-controller-manager
# 2. Recent logs
kubectl logs -n kube-system kube-controller-manager-<node-name> -f --tail=100

Normal Standard: Pod status Running; logs without controller failed type errors

Anomaly Handling: Restart component:

kubectl delete pod -n kube-system kube-controller-manager-<node-name>

(auto-recreated)

scheduler

Check Target: Scheduling function normal, no scheduling failures

Check Commands:

# 1. Pod status
kubectl get pods -n kube-system -l component=kube-scheduler
# 2. Scheduler events
kubectl get events --sort-by='.lastTimestamp' | grep -i scheduler

Normal Standard: Pod status Running; no FailedScheduling events (non-resource-insufficiency caused)

1.2 Node Components (Cluster "Limbs")

Nodes are the carrier for Pods; need to check kubelet (node agent), container runtime (e.g., containerd), and system parameters.

1.2.1 Node Basic Status

# 1. View all node status (core: Ready)
kubectl get nodes -o wide

# 2. View node details (troubleshoot NotReady cause)
kubectl describe node <node-name> | grep -A 10 "Conditions"

Normal Standard : All nodes status Ready, Ready condition LastHeartbeatTime ≤10s from current; no MemoryPressure / DiskPressure / PIDPressure (resource pressure).

Anomaly Handling :

1.2.2 Container Runtime (containerd/cri-o)

# 1. Check containerd status (mainstream runtime)
systemctl status containerd --no-pager

# 2. Check container runtime communication with kubelet
crictl info | grep "runtimeName"  # confirm runtime correct
crictl pods  # list Pods normally (consistent with kubectl get pods)

Normal Standard : containerd status active (running); crictl commands error-free, Pod list consistent with kubectl.

Anomaly Handling :

1.2.3 Key System Parameters

# 1. Check if swap is off (K8s mandatory)
cat /proc/swaps | grep -v "Filename"  # no output = normal

# 2. Check kernel parameters (network forwarding, bridging)
sysctl net.ipv4.ip_forward net.bridge.bridge-nf-call-iptables

Normal Standard : swap off; both kernel parameters value = 1.

Anomaly Handling :

1.3 Network Components (Cluster "Blood Vessels")

Network components (CNI) handle Pod-to-Pod and Pod-to-external communication; common ones are flannel, calico; need to check component status and network connectivity.

1.3.1 CNI Component Status

# Using calico as example (flannel similar, label app=flannel)
kubectl get pods -n calico-system  # all Pods must be Running
kubectl get daemonset -n calico-system calico-node  # replica count equals node count

Normal Standard : calico-node (DaemonSet) replica count = node count, all Pods status Running; no CrashLoopBackOff.

1.3.2 Network Connectivity Verification

# 1. Cross-node Pod communication test (pick two Pods on different nodes)
kubectl exec -it <pod1-name> -n <ns1> -- ping -c 3 <pod2-ip>

# 2. Check Service and Endpoint match (ensure Service can find Pods)
kubectl get svc <svc-name> -n <ns> -o jsonpath='{.spec.clusterIP}'  # view Service IP
kubectl get endpoints <svc-name> -n <ns>  # Endpoint must contain Pod IP:Port

Normal Standard : Cross-node Pods pingable; Service Endpoint non-empty, IP matches backend Pods.

Anomaly Handling :

1.4 Storage Components (Cluster "Warehouse")

Storage components handle data persistence; need to check PV/PVC binding status, CSI drivers (e.g., local-path-provisioner, longhorn), and storage backend health.

# 1. Check PVC status (core: Bound)
kubectl get pvc --all-namespaces | grep -v "Bound"  # no output = normal

# 2. Check PV status (must match PVC, no idle abnormal PVs)
kubectl get pv | grep -v "Available\|Bound"  # no `Failed`/`Released` status

# 3. Check CSI driver (using local-path as example)
kubectl get pods -n local-path-storage  # provisioner Pod must be Running

Normal Standard : All PVCs status Bound; PV status Available (unused) or Bound (used); CSI driver Pods no restarts.

Anomaly Handling :

2. Resource Usage Inspection Checklist

Resource usage imbalance is a primary cause of self-hosted cluster failures (e.g., OOM killing Pods, disk full causing component crashes); need to investigate from "Node - Pod - Storage" three layers.

2.1 Node Resource Monitoring (Overall Capacity)

Focus on CPU, memory, disk "usage" and "saturation"; avoid only watching usage while ignoring actual load (e.g., CPU saturation better reflects busyness).

# 1. Node resource overview (kubectl top)
kubectl top node  # view CPU/memory usage

# 2. Disk space and inode check (avoid disk full/inode exhaustion)
df -h | grep -E "/$|/var/lib/docker|/data"  # key partition usage ≤85%
df -i | grep -E "/$|/var/lib/docker|/data"  # inode usage ≤85%

# 3. (Optional) PromQL for CPU saturation (requires Prometheus)
# avg(rate(node_cpu_saturation_ratio[5m])) by (node)  # saturation ≤80% normal

Normal Standard :

Anomaly Handling :

2.2 Pod Resource Investigation (Individual Load)

Pod resource anomalies directly affect business; focus on "high usage", "OOM events", "Pending status" three problem types.

2.2.1 High Resource Usage Pods

# 1. High CPU usage Pods (Top 10)
kubectl top pod --all-namespaces --sort-by='cpu' | tail -n 11

# 2. High memory usage Pods (Top 10)
kubectl top pod --all-namespaces --sort-by='memory' | tail -n 11

# 3. View Pod resource limits (judge if configured reasonably)
kubectl get pods <pod-name> -n <ns> -o jsonpath='{.spec.containers[0].resources}'

Normal Standard : Pod actual resource usage ≤ configured limits; no long-term (≥1h) Pods exceeding limits 90%.

Anomaly Handling :

2.2.2 OOM and Abnormal Restart Pods

# 1. View last 24h OOM events (node level)
journalctl -u kubelet --since "24h ago" | grep -i "out of memory"

# 2. View Pod restart counts (restart >0 needs attention)
kubectl get pods --all-namespaces -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.namespace}{"\t"}{.status.containerStatuses[0].restartCount}{"
"}{end}' | grep -v "0$"

Normal Standard : No OOM events in 24h; core business Pod restart count =0, non-core Pod restart count ≤1.

Anomaly Handling :

2.2.3 Pending Status Pods

# 1. List all Pending Pods
kubectl get pods --all-namespaces | grep "Pending"

# 2. Troubleshoot Pending reason (focus on Events)
kubectl describe pod <pending-pod> -n <ns> | grep -A 20 "Events"

Common Causes :

2.3 Namespace Resource Quotas

If cluster shared by multiple teams, need to check namespace resource quota usage to prevent single team exhausting cluster resources.

# 1. View namespace quota configuration
kubectl get resourcequota --all-namespaces

# 2. View specific namespace quota usage (using production as example)
kubectl describe resourcequota <quota-name> -n production

Normal Standard : Each namespace resource usage (CPU/memory/PVC) ≤ quota 80%, reserving buffer space.

Anomaly Handling :

3. Inspection Tools and Automation Suggestions

Manual command execution inefficient; recommend combining tools for "automated inspection + visual monitoring" to reduce operational cost.

3.1 Basic Toolset

Command-line tools : kubectl (core), crictl (container runtime), etcdctl (etcd management), kube-ps1 (terminal shows current cluster/namespace).

Visualization tools : Prometheus + Grafana (metrics visualization, recommended import panels: Node Exporter Full (ID:1860), K8s Cluster Resource Usage (ID:893)).

Alerting tools : Alertmanager (integrates with Prometheus, auto-alerts on resource threshold exceedance or component anomalies, e.g., CPU saturation >90% sends email).

3.2 Automated Inspection Script Example

Encapsulate core checks into script, run daily via crontab, output inspection report.

#!/bin/bash
# Self-hosted K8s cluster automated inspection script
REPORT_DATE=$(date +%Y-%m-%d_%H-%M-%S)
REPORT_FILE="/tmp/k8s_inspect_$REPORT_DATE.log"

# Initialize report
echo "=== Self-hosted K8s Cluster Inspection Report ($REPORT_DATE)===" > $REPORT_FILE

# 1. Control plane component status
echo -e "
1. Control Plane Component Status:" >> $REPORT_FILE
kubectl get pods -n kube-system -l tier=control-plane >> $REPORT_FILE 2>&1

# 2. Node status
echo -e "
2. Node Status (NotReady needs attention):" >> $REPORT_FILE
kubectl get nodes | grep -v "Ready" >> $REPORT_FILE 2>&1
if [ $? -ne 0 ]; then
  echo "All nodes are Ready" >> $REPORT_FILE
fi

# 3. PVC status (non-Bound needs attention)
echo -e "
3. PVC Status (non-Bound list):" >> $REPORT_FILE
kubectl get pvc --all-namespaces | grep -v "Bound" >> $REPORT_FILE 2>&1
if [ $? -ne 0 ]; then
  echo "All PVCs are Bound" >> $REPORT_FILE
fi

# 4. High memory usage Pods (Top 5)
echo -e "
4. High Memory Usage Pods (Top 5):" >> $REPORT_FILE
kubectl top pod --all-namespaces --sort-by='memory' | tail -n 6 >> $REPORT_FILE 2>&1

# 5. Disk usage (over 85% needs attention)
echo -e "
5. Key Partition Disk Usage (over 85% list):" >> $REPORT_FILE
df -h | grep -E "/$|/var/lib/docker|/data" | awk '$5 ~ /[8-9][5-9]%/ || $5 ~ /100%/' >> $REPORT_FILE 2>&1
if [ $? -ne 0 ]; then
  echo "All key partitions usage ≤85%" >> $REPORT_FILE
fi

# Output report path
echo -e "
Inspection report saved to: $REPORT_FILE"

Usage : Save as k8s_inspect.sh, add execute permission ( chmod +x k8s_inspect.sh), configure crontab ( 0 9 * * * /path/to/k8s_inspect.sh, daily 9 AM execution).

3.3 Visualization Monitoring Supplement

Configure "Cluster Health Dashboard" in Grafana for real-time display:

Component status: control plane/Pod/CNI/CSI uptime (target 100%).

Resource metrics: node CPU/memory/disk usage, Pod restart count, OOM event count.

Alert panel: unresolved alerts (e.g., node NotReady, PVC Pending).

4. Inspection Cycle and Issue Closure

4.1 Inspection Frequency Recommendations

Daily Inspection : Once daily (automated script execution, focus on anomaly items).

Deep Inspection : Once weekly (manual check certificate validity, etcd data backup, kernel parameters, network routing).

Ad-hoc Inspection : When cluster failures occur (e.g., large-scale Pod restarts, service inaccessible), temporarily execute full check suite.

4.2 Issue Recording and Tracking

Establish "Inspection Issue Ledger" recording:

Issue phenomenon (e.g., "node-01 memory usage 92%").

Handling steps (e.g., "clean log collection Pod, restart core services").

Result verification (e.g., "memory usage dropped to 75%, service restored").

Preventive measures (e.g., "add resource limits for log Pod, configure log rotation").

Through closed-loop management, avoid recurring issues, gradually improve cluster stability.

Summary

Self-hosted K8s cluster inspection core is "early detection, early handling": component status inspection ensures "skeleton doesn't collapse", resource usage inspection avoids "blood flow blockage". This checklist covers 80%+ common risk points; recommend adjusting checks based on actual cluster scale (node count, business type) — small clusters can simplify to "component status + node resources" two items, large clusters need supplement "namespace quota + storage backend health" checks.

Long-term, automation (scripts + monitoring alerts) is key to reducing inspection cost, while issue closure is core to improving cluster stability; combining both enables self-hosted K8s clusters to run continuously and reliably.