Kubernetes Containerization: Monitoring, Logging, Sidecar Injection & Custom PodIP

Introduction

The author, a containerization engineer at a listed internet company, explains why containerization is essential for cost reduction and operational efficiency, and outlines common pitfalls that can lead to wasted investment.

Monitoring

Three monitoring categories are described: metric monitoring, business monitoring, and tracing. Metric monitoring relies on Prometheus with Kubernetes service discovery. The author notes that Prometheus storage can become a bottleneck due to WAL size, and evaluates InfluxDB and TiDB for remote read/write, recommending external deployment to avoid high memory usage.

Logging

Logging is divided into std logs (collected from Docker data directories via a DaemonSet running Fluentd) and file logs (collected by a Filebeat sidecar after sharing a directory via emptyDir). Both log types are forwarded to an ELK stack.

Continuous Delivery Integration

Helm is used as the bridge between continuous delivery pipelines and Kubernetes, converting deployment configurations into JSON objects and providing built‑in resource status monitoring.

Sidecar Injection

To automate Filebeat sidecar deployment, a Mutating Admission Webhook is employed. The webhook intercepts pod creation, applies a JSON Patch to inject the sidecar based on annotations, and can also enforce security policies such as hostPort or probe validation.

Custom PodIP

For services like Redis that require stable IPs, the team uses Calico CNI with the annotation cni.projectcalico.org/ipAddrs. Because pods are created from higher‑level templates, a second Mutating Admission Webhook adds the annotation to each pod in a StatefulSet, enabling deterministic IP assignment without custom IPAM development.

Task Scheduling

Legacy PHP workloads used Kubernetes CronJobs, which caused log visibility issues, delayed starts, and cluster pressure. The solution was to adopt the open‑source goCron library, deploying a dedicated Deployment for scheduling and communicating via gRPC with a custom target field in the proto definition.

Cluster Event Monitoring

Since Kubernetes events expire after one hour, the team built a middleware that watches all namespace events, forwards them to Elasticsearch, aggregates them, and exposes metrics to Prometheus, providing a longer‑term health view.

Time and Resource Simulation

Containers share the host kernel, so /proc/cpuinfo and /proc/meminfo report host resources, misleading applications. Three mitigation strategies are discussed: passing resource limits to the runtime, enabling JVM flags (

-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap

), and rewriting kernel parameters via LXCFS mounted with hostPath. For accurate time, libfaketime is used with LD_PRELOAD and FAKETIME environment variables.

Etcd Data Recovery

When the API server is unavailable, raw etcd data (stored as protobuf) can be restored to YAML using the open‑source Auger project.