Why Can’t a Kubernetes Namespace Delete? A Deep Dive into the Terminating State

Kubernetes Namespace is a logical container that groups resources within a cluster. When a Namespace is deleted, its resources must be cleaned up first; otherwise the Namespace may remain in a "Terminating" state.

Reproducing the Issue

To study the problem, a test Namespace tobedeletedb was created. Logs from the API Server showed a deletion request from kubectl (source IP 192.168.0.41), but the Namespace never fully disappeared.

Controller Manager Behavior

Enabling the highest log level on the Kube Controller Manager revealed two log streams:

Records of the Namespace deletion request.

Repeated attempts by the Namespace Controller to fetch information about tobedeletedb.

The controller kept trying to clean up resources inside the Namespace, indicating that it could not simply delete the logical container without first removing the referenced resources.

API Groups and Versions

Kubernetes APIs are organized by group and version, allowing independent evolution. For example, an Ingress resource uses the group networking.k8s.io and version v1beta1:

kind: Ingress
metadata:
  name: test-ingress
spec:
  rules:
  - http:
      paths:
      - path: /testpath
        backend:
          serviceName: test
          servicePort: 80

Every resource belongs to a specific apiVersion, and listing resources requires querying the available group/version list.

Why the Controller Fails to Delete

The Namespace Controller attempted to retrieve the API group/version list and failed when accessing metrics.k8s.io/v1beta1. The API Server returned the error "the server is currently unable to handle the request".

API Server Extension Mechanism

The API Server can delegate calls to extension services via an apiservice. The Metrics Server implements the metrics.k8s.io/v1beta1 API, and the apiservice definition links this group/version to the Metrics Server service.

Network Connectivity Issue

In the Alibaba Cloud Kubernetes environment, the API Server runs on the host network (ECS) while the Metrics Server runs on the pod network. Communication between them depends on VPC routing.

Missing VPC route entries prevented the API Server (IP 192.168.0.193) from reaching the Metrics Server (IP 172.16.1.12), causing the metrics API calls to fail.

Route Controller Failure

The Cloud Controller Manager includes a Route Controller that watches node status and VPC route tables, automatically adding missing routes. Logs showed the controller could not retrieve VPC instance information because the RAM role policy for VPC access had been altered.

After correcting the RAM policy (changing Effect to Allow), the missing routes were restored and all Terminating Namespaces disappeared.

Overall Diagnosis

The issue involved six cluster components: API Server, Metrics Server (as an API extension), Namespace Controller, Route Controller, VPC routing, and RAM role authorization. The root cause was a network connectivity failure between API Server and Metrics Server due to a missing VPC route, compounded by an incorrect RAM role policy that prevented the Route Controller from repairing the route.

By restoring the VPC route and fixing the RAM policy, the Namespace deletion process completed successfully.