Build a Full Kubernetes DevOps Pipeline: From Containerization to Monitoring

Case Background

A company built a micro‑service‑based application and chose Kubernetes for container orchestration to accelerate feature delivery, ensure high availability, and automate CI/CD and continuous monitoring.

Technology Stack

Code Management: GitHub / GitLab

CI/CD Tools: Jenkins, GitLab CI, ArgoCD

Containerization: Docker

Orchestration: Kubernetes (K8s)

Service Mesh (optional): Istio or Linkerd

Monitoring: Prometheus + Grafana

Log Management: ELK / EFK (Elasticsearch, Fluentd, Kibana)

Practical Steps

1. Application Containerization

The development team splits the application into independent micro‑services, each built into a Docker image.

Write a Dockerfile for each service, for example:

FROM python:3.9-slim
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
CMD ["python", "app.py"]

Build and push the image: docker build -t my-registry/my-app:latest . then docker push my-registry/my-app:latest.

2. Configure Kubernetes Deployment Files

Create Deployment and Service manifests to define how each micro‑service runs in the cluster.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: my-registry/my-app:latest
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: my-app-service
spec:
  selector:
    app: my-app
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  type: LoadBalancer

3. Build CI/CD Pipeline

Configure CI to run unit tests, code quality checks, and build Docker images on each commit. CD automatically deploys the new image to the Kubernetes cluster.

stages:
  - build
  - deploy

build:
  stage: build
  script:
    - docker build -t my-registry/my-app:latest .
    - docker push my-registry/my-app:latest

deploy:
  stage: deploy
  script:
    - kubectl apply -f k8s/deployment.yaml

4. Introduce Monitoring and Logging

Monitoring: Deploy Prometheus to scrape cluster and application metrics, then configure Grafana dashboards for visualization.

Logging: Deploy an EFK stack; Fluentd forwards container logs to Elasticsearch, and Kibana is used for log analysis.

5. Optional Service Mesh

If advanced traffic management or security is required, introduce Istio (or Linkerd) to enable blue‑green deployments, canary releases, and fine‑grained access control.

6. Failure Drills and Continuous Optimization

Use tools such as Chaos Mesh to simulate failures and verify system resilience.

Analyze monitoring data to adjust resource allocations and improve service performance continuously.