Improving Cluster Stability: CI/CD, Monitoring, Logging, Documentation, and Request Routing Solutions

The company's clusters were constantly on the brink of failure; after three months of investigation, the main reasons identified were an unstable release process, absence of a monitoring platform (the most critical factor), missing logging system, severe lack of operational documentation, and unclear request routing.

Unstable Release Process – The solution is to rebuild the release workflow by fully containerizing services on Kubernetes and establishing a CI/CD pipeline centered on Kubernetes. Developers push code to the developer branch, which is merged into environment‑specific branches, triggering alerts via WeChat, launching a GitLab‑Runner pod in the cluster, and executing test, image build, and pod update steps. Security‑sensitive resources such as ConfigMaps and Secrets are managed through Rancher rather than being stored in the code repository.

Monitoring Platform – Build a reliable, federated monitoring system based on Prometheus, supplemented by Shell/Go scripts and Sentry for alerting via WeChat or email. The platform monitors OS‑level, application‑level, and business‑level metrics across multiple clusters from a single visual interface, using Kong as the ingress for traffic monitoring.

Logging System – Deploy an Elasticsearch‑based log collection solution to capture and retain logs from Kubernetes pods, addressing the difficulty of accessing logs after pod restarts. The design supports remote storage or host‑mounted logs and provides searchable, visualized log data for troubleshooting.

Operational Documentation – Create a documentation center (e.g., using Yuque) that records operational procedures, scripts, and issue resolutions, ensuring that all maintenance steps are documented, version‑controlled, and accessible to authorized personnel.

Request Routing – Redesign traffic flow by integrating Kong and Istio for authentication, authorization, and proxying, establishing clear north‑south and east‑west routing paths, and visualizing the request routes to contain failures.

In summary, by constructing a Kubernetes‑centric CI/CD pipeline, a Prometheus‑based federated monitoring platform, an Elasticsearch logging system, a Yuque documentation hub, and a Kong/​Istio unified traffic architecture, the company can achieve high availability and stability for its services on Kubernetes clusters.