Why Does My ip‑Target ALB Ingress Show Unhealthy Pods? Fix Security Groups in EKS

Problem Overview

When an AWS EKS cluster uses the ALB Ingress Controller with target-type: ip, traffic is sent directly to pod IPs, bypassing NodePort. A frequent failure mode is that the pods appear unhealthy in the ALB target group even though they are reachable from inside the cluster.

Root Cause: Security‑Group Path from ALB to Pod IP

The failure is usually caused by security‑group rules that block the health‑check traffic originating from the ALB nodes.

How ALB Health Checks Relate to Security Groups

Health‑Check Origin – ALB Nodes

When an ALB is created, AWS launches load‑balancer nodes in the selected VPC subnets. The health‑check requests are sent from the IP addresses of these nodes.

Security‑Group Roles

SG_ALB : Security group attached to the ALB nodes; its outbound rules control what the health‑check packets can reach.

SG_POD : Security group attached to the ENI that backs each pod when target-type: ip is used; its inbound rules decide which sources may reach the pod.

A successful handshake requires SG_ALB to allow outbound probes and SG_POD to allow inbound traffic from SG_ALB.

Network Security for ip -Target Pods

Pod ENI‑Attached Security Group (SG_POD)

Using target-type: ip implies the cluster runs the AWS VPC CNI plugin, which assigns real VPC IPs to pods via ENIs. The security group is attached directly to the pod ENI.

Factors Influencing SG_POD

Security Groups for Pods (recommended) : Assigns a dedicated security group to each pod via pod annotations or a SecurityGroupPolicy. The ALB Ingress Controller automatically adds an inbound rule that references SG_ALB.

Worker‑node security group (legacy) : If Security Groups for Pods are not used, the pod may inherit the worker node’s security group, which is less granular.

SG_ALB outbound rules : Must permit traffic to the pod’s service port and health‑check port; by default they allow all outbound traffic.

Practical Configuration Steps

Identify the Relevant Security Groups

SG_ALB : View in the AWS console under EC2 → Load Balancers → select the ALB → “Security” tab.

SG_POD : When using Security Groups for Pods, specify it via pod annotation or SecurityGroupPolicy. It can also be discovered with kubectl describe pod <pod-name> -n <namespace> (check annotations) or by searching the pod IP in EC2 → Network Interfaces.

Configure SG_POD Inbound Rules (Core)

Source : Set to the SG_ALB ID (e.g., sg‑alb11111), not a raw IP.

Protocol : Match the service/health‑check protocol (typically TCP).

Port range : Include both the service port and the health‑check port (e.g., 8080).

Description : Optional, e.g., “Allow health checks and traffic from ALB”.

Example : If SG_ALB = sg‑alb11111, SG_POD = sg‑pod22222, and the service/health‑check port is 8080, add the following inbound rule to sg‑pod22222:

Type: Custom TCP

Protocol: TCP

Port range: 8080

Source: sg‑alb11111

Check SG_ALB Outbound Rules

By default, ALB security groups allow all outbound traffic. If they have been restricted, ensure they permit traffic to the pod IP and the health‑check port.

Verify Network ACLs (NACLs)

ALB subnet NACLs : Outbound rules must allow traffic to the pod subnet; inbound rules must allow return traffic.

Pod subnet NACLs : Inbound rules must allow traffic from the ALB subnet; outbound rules must allow the response.

Because NACLs are stateless, both directions need explicit rules.

Tip : If the ALB Ingress Controller has sufficient IAM permissions, it will automatically create most SG_POD inbound rules, but you should still know how to inspect and fix them manually.

Common Pitfalls and Debugging Techniques

Typical Mistakes

Hard‑coding ALB node IPs as the source in SG_POD (use SG_ALB ID instead).

Misconfiguring the target‑group health‑check (wrong path, port, or expected HTTP status).

Overly restrictive NACL rules.

Insufficient IAM permissions for the ALB Ingress Controller to modify security groups.

Debugging Steps

Check ALB Ingress Controller logs:

kubectl logs -n kube-system -l app.kubernetes.io/name=aws-load-balancer-controller

In the AWS console, review the target‑group health‑check settings and target status.

Inspect the load balancer listeners under EC2 → Load Balancers.

Use VPC Flow Logs to filter traffic from ALB node IPs to pod IPs and look for REJECT entries.

Run kubectl describe on the pod, service, and endpoints to verify IPs, ENIs, and attached security groups.

From inside the pod, test the service endpoint:

kubectl exec -it <pod-name> -n <namespace> -- curl -v http://localhost:<container-port><health-check-path>

Conclusion

Health checks originate from ALB nodes and target pod IPs.

SG_POD inbound rules must reference SG_ALB as the source.

Use Security Groups for Pods for fine‑grained control.

Rely on the ALB Ingress Controller for automation, but be prepared to troubleshoot manually.