Tagged articles
kubelet
71 articles · Page 1 of 1
This guide details diagnosing and resolving Kubernetes node DiskPressure alerts by analyzing filesystem signals (nodefs, imagefs, containerfs), eviction mechanisms, image garbage collection, log rotation, ephemeral storage limits, kubelet configuration, monitoring strategies, and recovery procedures with practical commands for containerd/CRI-O environments.
ContainerdDiskPressureKubernetes
0 likes · 30 min read

This article explains Kubernetes volume mounting for ConfigMaps and Secrets, detailing how directory mounts achieve hot reloads via atomic symlink switching, why subPath mounts never update, and how projected volumes merge multiple configuration sources into a single directory.
ConfigMapHot ReloadKubernetes
0 likes · 10 min read

This article explains Kubernetes architecture, covering the master-node model, core components like API server, etcd, scheduler, controller manager, kubelet, and kube-proxy, and how they coordinate to deploy and manage containerized applications.
API ServerK8sKubernetes
0 likes · 6 min read

This guide walks Kubernetes operators through a systematic, step‑by‑step process for diagnosing nodes stuck in the NotReady state, covering kubelet status reporting, common failure reasons, detailed command‑line checks, root‑cause analysis, remediation steps, verification, and long‑term preventive measures.
KubernetesNotReadycertificate
0 likes · 42 min read

This article offers a free 172‑page Alibaba Cloud Kubernetes handbook that explains six core principles and typical issues, and provides a detailed technical analysis of a recurring node NotReady problem, covering kubelet roles, PLEG timing, and the three‑minute readiness window.
Alibaba CloudCluster troubleshootingKubernetes
0 likes · 7 min read

The article explains Kubernetes pod scheduling in detail, covering how the kube‑scheduler filters and scores nodes, binds the pod, and how kubelet launches containers, plus common reasons for pods staying Pending and useful troubleshooting commands.
KubernetesNodeSelectorPod Scheduling
0 likes · 8 min read

This comprehensive guide walks Kubernetes operators through a step‑by‑step process for diagnosing node health problems—such as NotReady, MemoryPressure, DiskPressure, PIDPressure, and NetworkUnavailable—by examining node conditions, reviewing events, checking system resources, inspecting component logs, applying targeted fixes, and verifying recovery, all illustrated with real‑world commands and examples.
CNIDiskPressureKubernetes
0 likes · 44 min read

This article walks Kubernetes operators through a systematic investigation of NotReady node symptoms, explaining the kubelet status mechanism, detailing each diagnostic step—from verifying node conditions with kubectl to checking kubelet, container runtime, resources, network, and certificates—and providing concrete remediation and preventive measures.
ContainerdKubernetesNotReady
0 likes · 35 min read

This article dissects the Kubelet component of Kubernetes, detailing its Go‑based architecture, core responsibilities, event‑driven syncLoop, PodWorkers concurrency model, syncPod creation flow, PLEG health monitoring, and provides practical debugging commands for production environments.
ConcurrencyDebuggingGo
0 likes · 14 min read

This article walks readers through Kubernetes’s fundamental architecture by dissecting its Go source code, explaining key concepts such as the API server, controllers, informers, the control loop, Kubelet, and extensibility mechanisms like CRDs and admission webhooks, complete with illustrative diagrams and code snippets.
CRDControllerGo
0 likes · 11 min read

Kubernetes 1.33 introduces in‑place vertical pod resizing, allowing administrators to adjust CPU and memory resources on running containers without restarting pods, reducing downtime for stateful workloads, improving cost efficiency, and integrating with VPA, while outlining implementation details, supported runtimes, limitations, and practical demos.
In‑Place Vertical ScalingKubernetesPod Resizing
0 likes · 18 min read

This guide walks you through preparing three AlmaLinux servers, disabling firewalls and SELinux, installing Containerd as the CRI, adding Kubernetes repositories, installing kubeadm, kubelet and kubectl, configuring the runtime, and verifying each component so you can confidently bootstrap a production‑ready Kubernetes cluster.
AlmaLinuxContainerdKubernetes
0 likes · 21 min read

This article examines a recurring Kubernetes issue where a node becomes NotReady for exactly three minutes after a kubelet restart, explains the underlying PLEG and node‑status mechanisms, and offers a free, in‑depth Kubernetes handbook for further study.
KubernetesNode NotReadyPLEG
0 likes · 7 min read

This article explains the design of Kubernetes 1.32 Device Resource Allocation (DRA), detailing the four new CRDs, the roles of kube‑controller‑manager, kube‑scheduler and kubelet plugins, and the required admission webhook and RPC interfaces for managing device resources.
CRDDRADeviceResourceAllocation
0 likes · 8 min read

In a Kubernetes cluster the kube-scheduler pod may stay pending due to a container-name reservation conflict; this guide shows how to locate the offending container with ctr, delete it, restart kubelet, and verify that the pod returns to Running state.
CTRContainerdKubernetes
0 likes · 5 min read

This article explains how Kubernetes, through the Kubelet and CRI components such as containerd and cri‑o, pulls container images, stores them on the node, and performs periodic garbage collection based on configurable age and disk‑usage thresholds.
Image Garbage CollectionKubernetescontainer-runtime
0 likes · 6 min read

This article dissects Kubelet's Pod admission pipeline, explaining how syncLoopIteration gathers pod data, how HandlePodAdditions invokes canAdmitPod, and how six registered admit handlers—Eviction, System Allowlist, Resource Allocation, Predicate, AppArmor, and Shutdown—evaluate each pod with concrete code examples and decision logic.
Admission HandlersGoKubernetes
0 likes · 14 min read

This article dissects Kubelet's ImageManager and ImageGCManager, detailing their interfaces, implementation structs, pull‑policy handling, parallel image pulling, GC policy thresholds, and the step‑by‑step execution flow that governs image lifecycle on a node.
ContainerRuntimeGarbageCollectionGo
0 likes · 7 min read

When a new Pod is created, Kubelet coordinates with the CRI and CNI components to set up the sandbox, configure networking, pull images, create and start containers, using gRPC calls and command‑line interactions, with details varying across container runtimes such as containerd, cri‑o, and Docker.
CNICRIKubernetes
0 likes · 5 min read

This article explains how Kubernetes uses the device‑plugin framework to extend resources beyond CPU and memory, details the kubelet registration and allocation workflow, and walks through the relevant source code in pkg/kubelet/cm/devicemanager that builds the OCI spec.
CDIDRADevice Plugin
0 likes · 5 min read

The article dissects Kubelet’s VolumeManager, detailing its asynchronous loops, the VolumeManager interface, how it is started from Kubelet.Run, the handling of Attach/Mount and Unmount operations during pod sync, the internal struct fields, and the plugin initialization process that together manage the full lifecycle of pod volumes.
GoKubernetesPod Lifecycle
0 likes · 10 min read

This article dissects Kubelet's core data structures—podManager, podWorkers, and workQueue—explaining how they store and synchronize regular pods, static pods, and mirror pods, and detailing the lifecycle steps for pod creation, deletion, and cleanup.
KubernetesMirrorPodWorkqueue
0 likes · 10 min read

The article explains that the apparent probeCh in kubelet is actually three separate channels—livenessCh, readinessCh, and startupCh—managed by livenessManager, readinessManager, and startupManager, details the ProbeManager implementation that creates probe workers via AddPod, and shows how syncLoopIteration processes probe updates to adjust pod status.
GoKubernetesProbe
0 likes · 8 min read

The article dissects kubelet's podUpdateCh, explaining how it ingests pod data from file, URL, and the API server, the Linux‑only file watch, periodic full scans, merging logic, and the underlying code paths that push updates into the channel.
KuberneteskubeletpodUpdateCh
0 likes · 7 min read

The article dissects the kubelet architecture, detailing its main syncLoop control cycle, auxiliary loops, and key managers such as podManager, podWorkers, evictionManager, probeManager, and runtime components, while explaining how pod updates, PLEG mechanisms, and various channels coordinate pod lifecycle and resource handling.
KubernetesRuntimecloud-native
0 likes · 9 min read

This article walks through diagnosing and resolving two common Kubernetes issues—node memory leaks caused by kmem accounting and expired cluster certificates—by showing how to detect the problems, rebuild runc and kubelet, and extend certificate validity using kubeadm and manifest edits.
KubernetesMemory Leakcertificate renewal
0 likes · 12 min read

This article examines Kubernetes’ Kubelet component through an object‑oriented lens, detailing its role, key responsibilities, abstract properties and methods, and illustrating implementation steps such as resource checks and pod scheduling, to show how OO abstraction clarifies complex system behavior.
KubernetesObject-Oriented Designcloud-native
0 likes · 5 min read

This guide explains why old Kubernetes clusters that use Docker can run out of disk space, describes the symptoms such as pods stuck in ContainerCreating, and provides step‑by‑step commands to clean Docker files, prune images, adjust kubelet settings, and prevent future disk‑full problems.
Operationsdisk cleanupgarbage collection
0 likes · 11 min read

This article explains how CoreDNS runs on a Caddy‑based HTTP/2 server in Kubernetes, how kubelet injects the cluster DNS IP into each container’s /etc/resolv.conf, and how different dnsPolicy settings (Default, ClusterFirst, ClusterFirstWithHostNet, None) affect the resolv.conf configuration, including key options and examples.
CoreDNSKubernetesdnsPolicy
0 likes · 6 min read

Learn step‑by‑step how to identify and clean up disk‑space exhaustion in older Kubernetes clusters using Docker, including manual log removal, node draining, Docker restart, image pruning, and configuring kubelet garbage‑collection parameters to prevent future outages.
Dockerdisk cleanupgarbage collection
0 likes · 10 min read

This article walks through a real‑world Kubernetes incident where a node’s disk usage exceeded the eviction threshold, causing pods to enter the Evicted state, and details the investigation steps, root‑cause analysis, and practical remediation actions.
AWSDiskPressureKarpenter
0 likes · 6 min read

This article walks through the complete Kubernetes workflow from a user‑submitted Deployment request to the creation and scheduling of the resulting Pod, detailing the roles of the control‑plane components, node services, admission webhooks, and the various plugins involved.
KubernetesPodcloud-native
0 likes · 7 min read

Learn the critical Kubernetes security measures for production environments, including RBAC access control, network policies, secret management, continuous monitoring, patch updates, API server hardening, Kubelet protection, pod security policies, and container hardening techniques, each illustrated with practical YAML examples and command snippets.
ContainerHardeningKubernetesNetworkPolicy
0 likes · 10 min read

This article walks through the complete Kubernetes pod creation workflow, from submitting the YAML with kubectl to the API server, storing the definition in etcd, scheduling, kubelet orchestration, container runtime delegation, CNI networking, health probing, and endpoint setup for services.
CNIKubernetesPod Lifecycle
0 likes · 3 min read

A self‑built Kubernetes cluster was compromised when an unprotected node with empty iptables and a kubelet that allowed anonymous API access was hijacked for Monero mining, prompting a detailed post‑mortem, root‑cause analysis, and hardening recommendations.
crypto miningincident responsekubelet
0 likes · 5 min read

The article explains how Kubernetes allocates Pod IPs—from cluster and node CIDR configuration, through the kubelet‑driven Pod lifecycle and CNI IPAM process, including dual‑stack support, hostNetwork handling, and optional IP reservation/reclamation mechanisms for stable stateful workloads.
CNIDual StackIPAM
0 likes · 21 min read

When a pod crashes and continuously restarts, standard kubelet logs may miss the previous container's output, but using kubectl logs with the --previous flag lets you access the logs of the last terminated instance, as explained with commands, file locations, and practical verification steps.
Kuberneteskubectlkubelet
0 likes · 7 min read

When a pod crashes and continuously restarts, kubelet logs may miss the output, but using the kubectl --previous flag lets you access the logs of the prior container instance by reading the symlinked files under /var/log/pods, enabling effective troubleshooting of Kubernetes crashes.
cloud-nativekubeletlogs
0 likes · 7 min read

When a pod crashes and continuously restarts, using the standard kubelet logs may fail to capture output, but the kubectl --previous flag can retrieve logs from the prior container instance by accessing the kubelet‑managed log files under /var/log/pods.
kubectlkubeletlog troubleshooting
0 likes · 6 min read

A detailed post‑mortem explains how excessive cgroup files, kubelet's sys‑CPU usage, soft‑interrupt scheduling delays, and a buggy page‑free routine caused intermittent hundreds‑of‑milliseconds network latency in an Alibaba Cloud ACK cluster, and how targeted CPU binding and kernel patches resolved the issue.
Kubernetescgroupcloud-native
0 likes · 14 min read

The article recounts a Kubernetes cluster intrusion where a misconfigured kubelet allowed crypto‑mining, details the forensic steps taken—including empty iptables, kubelet API exposure, and commented‑out settings—and offers concrete hardening recommendations to prevent similar attacks.
Cluster HardeningKubernetesSecurity
0 likes · 5 min read

Since Kubernetes 1.25 the default container image registry moved from k8s.gcr.io to the multi‑provider CDN‑backed registry.k8s.io, improving download speed and resilience, but requiring clusters to update image references or mirror images for restricted environments.
Container ImagesKubernetescloud-native
0 likes · 8 min read

This article explains how upgrading a Kubernetes node's CPU while using the static CPU manager policy can cause kubelet startup failures, details the underlying CPU manager architecture and topology handling, and provides a simple two‑step recovery method.
CPU ManagerCPU topologyKubernetes
0 likes · 15 min read

This article explains how Kubernetes' kubelet performs container and image garbage collection, details the configurable thresholds and limits, and presents step‑by‑step experiments that demonstrate the impact of different settings on image and container cleanup behavior.
DockerKubernetesgarbage collection
0 likes · 17 min read

This article explains the key Kubernetes control‑plane components—including etcd, the API Server, Controller Manager, Scheduler, as well as worker‑node components like Kubelet, kube‑proxy, and the container runtime—detailing their roles, interactions, and the underlying mechanisms such as Raft consensus and admission control.
API ServerKubernetescontrol plane
0 likes · 10 min read

This article explains the core components of the Kubernetes control plane—including etcd, the API Server, Controller Manager, Scheduler—as well as key worker‑node components like Kubelet, kube‑proxy, and the container runtime, detailing their roles, interactions, and essential functions.
API ServerKubernetescontrol plane
0 likes · 11 min read

This article explains the goals, design principles, and detailed components of Kubernetes architecture, covering its control plane, API server, etcd store, scheduler, kubelet, container runtime, and kube-proxy, and summarizes how these parts work together to provide a scalable, portable, and automated container orchestration platform.
Kubernetescontainer orchestrationcontrol plane
0 likes · 12 min read

A Kubernetes node was compromised for Monero mining due to empty iptables, an exposed kubelet API, and a mis‑commented configuration, prompting a detailed post‑mortem and practical hardening steps to prevent similar attacks.
KubernetesSecuritycrypto mining
0 likes · 5 min read

This article walks through diagnosing and resolving two frequent Kubernetes problems—memory‑leak errors that cause "cannot allocate memory" or "no space left on device" messages, and expired cluster certificates—by checking cgroup stats, recompiling runc and kubelet, and renewing certificates with kubeadm for long‑term validity.
KubernetesMemory Leakcertificate renewal
0 likes · 12 min read

A Kubernetes cluster was compromised when a misconfigured kubelet allowed anonymous API access, enabling attackers to run a Monero miner; the post details the investigation, root causes, and practical hardening steps to prevent similar intrusions.
Kubernetescrypto miningfirewall
0 likes · 5 min read

This article explains why container workloads suffer from CPU throttling and NUMA‑related performance loss in cloud‑native environments, examines Kubelet's CPU allocation policies, demonstrates the impact of CPU bursts and topology‑aware scheduling, and shows how Alibaba Cloud ACK mitigates these issues with concrete data.
Alibaba Cloud ACKCPU BurstCPU throttling
0 likes · 11 min read

After discovering a compromised node in our self‑built Kubernetes cluster that was being used for Monero mining, we traced the breach to empty iptables rules and a misconfigured kubelet allowing anonymous API access, then outlined firewall hardening, network isolation, and secure kubelet practices to prevent future intrusions.
Mining AttackSecurityfirewall
0 likes · 6 min read

This article details the challenges, strategies, and step‑by‑step procedures for upgrading a 1,000‑node Kubernetes cluster from version 1.10 to 1.17 without service interruption, covering compatibility checks, in‑place versus replacement upgrades, container‑restart avoidance, pod eviction handling, and TCP connection issues.
CNCFKubernetesVersion Skew
0 likes · 22 min read

Vivo’s internet team performed a zero‑downtime, in‑place upgrade of a 1,000‑node Kubernetes cluster from v1.10 to v1.17 by analyzing changelogs, backporting fixes, adjusting kubelet hash validation, adding tolerations, ensuring node labels, and using staged binary rollout, completing the process in roughly ten minutes.
K8s migrationKubernetescloud-native
0 likes · 19 min read

A self‑built Kubernetes cluster suffered a crypto‑mining intrusion due to empty iptables and a misconfigured kubelet, prompting a detailed post‑mortem that outlines the symptoms, root‑cause analysis, and practical hardening steps to protect similar environments.
crypto miningfirewallincident response
0 likes · 5 min read

This article explains the internal components of the kubelet, details the CRI architecture, outlines the roles of the Runtime and Image services, and provides code examples of the gRPC API definitions, helping readers grasp how Kubernetes manages containers and images.
CRIKubernetesgRPC
0 likes · 22 min read

After discovering a suspicious process on one of our self‑built Kubernetes nodes, we traced the intrusion to a misconfigured kubelet that exposed the API, allowing attackers to run a Monero mining script, and we outline the investigation steps and hardening measures to prevent similar breaches.
KubernetesSecuritycrypto mining
0 likes · 6 min read

A recent incident revealed that a misconfigured kubelet and missing firewall rules allowed an attacker to hijack a node in a self‑built Kubernetes cluster for Monero mining, prompting a detailed post‑mortem and a set of hardening recommendations.
KubernetesSecuritycontainer
0 likes · 7 min read

A Kubernetes node was compromised for Monero mining due to empty iptables, exposed kubelet API, and a commented‑out security flag, prompting a detailed forensic analysis and a set of hardening steps to secure the cluster against similar attacks.
Cluster HardeningSecuritycrypto mining
0 likes · 5 min read

This article walks through every component that processes a simple `kubectl run` command, from the client’s request parsing, through API‑server authentication, authorization, admission control, controller manager scheduling, to kubelet’s CRI and CNI handling that finally creates the pod and its containers.
CNICRIKubernetes
0 likes · 57 min read

This article walks through the complete internal journey of a `kubectl run nginx --image=nginx --replicas=3` command, detailing how the request is validated, authenticated, authorized, processed by the API server, passed through initializers, scheduled, and finally materialized as running pods by kubelet, with code excerpts from Kubernetes v1.21.
CNICRIKubernetes
0 likes · 62 min read

This article explores the Kubernetes kubelet source code, detailing its directory layout, the main kubelet.go entry point, the Run initialization sequence, the sync loop mechanics, and key questions about pod creation, synchronization, networking, volume management, and cleanup within a v1.13 cluster.
GoKubernetesPod Lifecycle
0 likes · 8 min read

This article provides a detailed, step‑by‑step tutorial for installing Kubernetes v1.16.0 on CentOS 7 virtual machines, covering Docker‑CE installation, prerequisite system configuration, master and node setup, flannel network plugin deployment, and includes all necessary command‑line snippets and the full kube‑flannel.yml manifest.
DockerFlannelKubernetes
0 likes · 20 min read

This article explains the kubelet’s key responsibilities, how it detects resource scarcity through eviction signals and thresholds, and provides practical guidance on configuring hard and soft eviction policies, reclamation strategies, and QoS‑based pod ranking to keep Kubernetes nodes stable.
KubernetesNode Managementcloud-native
0 likes · 12 min read

This article analyzes a recurring NotReady issue in Alibaba Cloud Kubernetes clusters, explaining how kubelet, NodeStatus, and the Pod Lifecycle Events Generator (PLEG) interact, why nodes become Ready for only three minutes after a kubelet restart, and how the underlying timeout mechanisms cause the problem.
Alibaba CloudKubernetesNode NotReady
0 likes · 7 min read


21CTOMay 19, 2020 · Cloud Native
This tutorial walks you through setting up a complete Kubernetes development environment on CentOS, covering prerequisite installations, Docker CE setup, kubelet/kubeadm/kubectl configuration, master node initialization, network add‑ons, node joining, and common troubleshooting tips.
CentOSKubernetescloud-native
0 likes · 14 min read

Learn how Kubernetes distinguishes compressible (CPU) and non‑compressible (memory, storage) resources, why default kubelet settings can cause resource contention, and how to use the Node Allocatable feature—configuring kube‑reserved, system‑reserved, and eviction thresholds—to safely reserve resources for system daemons and avoid cluster instability.
KubernetesNode Allocatablecgroups
0 likes · 9 min read

This article analyzes a Kubernetes cluster outage caused by a load‑balancer failure that prevented kubelet from connecting to the API server, explores the underlying HTTP/2 behavior, and presents debugging steps and code‑level fixes to restore reliable communication.
cloud-nativehttp2kubelet
0 likes · 13 min read

Kubernetes performs in‑place container upgrades by detecting spec changes, computing pod actions that selectively kill and restart only the affected containers while preserving the pod sandbox, allowing sidecar updates without full pod recreation, reducing downtime and enabling custom operators for gray‑scale rolling upgrades.
Container UpgradeKubernetesPod
0 likes · 16 min read

UK8S’s custom CNI plugin integrates VPC networking to give containers native cloud performance, but a bug caused kubelet to create a second sandbox container, leading to missing NETNS parameters and VPC IP leaks; the article details the investigation, root‑cause analysis, and the patch fixing the issue.
CNIIP leakKubernetes
0 likes · 15 min read

This article explains how Kubelet obtains pod configurations, processes updates through its syncLoop, coordinates pod workers, and ultimately invokes the container runtime's SyncPod method to keep node pods aligned with the desired state.
GoKubernetesPod Lifecycle
0 likes · 15 min read
