How to Fix Missing OIDC Provider in EKS for EBS CSI with Terraform

Case Background

When deploying an Apache RocketMQ cluster on EKS with Terraform, the broker requires persistent storage via an EBS volume. The StatefulSet uses volumeClaimTemplates to request the volume, but kubectl apply fails with an error indicating that no OpenID Connect (OIDC) provider was found for the cluster.

failed to provision volume with StorageClass "gp2": rpc error: code = Internal desc = Could not create volume "pvc-34f31fa2-a30e-4e2d-ba4e-c787d39791f6": could not create volume in EC2: operation error EC2: CreateVolume, get identity: get credentials: failed to refresh cached credentials, failed to retrieve credentials, operation error STS: AssumeRoleWithWebIdentity, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: cc63c0cd-155b-4053-ad62-b48c51906bc7, InvalidIdentityToken: No OpenIDConnect provider found in your account for https://oidc.eks.ap-southeast-1.amazonaws.com/id/8DE440F2238C6B959FB7C65EA52AF80B

The investigation revealed:

Cluster name: prod, region: ap-southeast-1, OIDC issuer URL:

https://oidc.eks.ap-southeast-1.amazonaws.com/id/8DE440F2238C6B959FB7C65EA52AF80B

.

No OIDC provider existed in AWS IAM ( aws iam list-open-id-connect-providers returned empty).

The EBS CSI driver ServiceAccount ebs-csi-controller-sa was bound to an IAM role eks-ebs-csi-driver, but could not obtain credentials.

The root cause is the missing OIDC provider, which prevents the IRSA mechanism from authenticating the driver.

What Is aws_iam_openid_connect_provider ?

This IAM resource registers an external identity provider that supports the OpenID Connect protocol. In EKS it links the cluster’s OIDC issuer with IAM, allowing Kubernetes ServiceAccounts to assume IAM roles via IRSA and obtain temporary AWS credentials.

How It Works

EKS OIDC Issuer : Each cluster generates a unique OIDC issuer URL that serves JWT tokens for ServiceAccounts.

Register with IAM : Using aws_iam_openid_connect_provider, the issuer URL is added to IAM as a trusted IdP.

IAM Role Binding : An IAM role is created with a trust policy that references the OIDC provider and a specific ServiceAccount (e.g., system:serviceaccount:kube-system:ebs-csi-controller-sa).

Permission Execution : Pods assume the role, receive temporary credentials from STS, and can call AWS APIs such as ec2:CreateVolume.

Why It Is Needed

The EBS CSI driver must call ec2:CreateVolume to provision storage. Without an OIDC provider, STS cannot validate the ServiceAccount’s JWT, so the driver cannot obtain credentials and the volume creation fails.

Configuring aws_iam_openid_connect_provider in Terraform

The following steps add the OIDC provider and fix the deployment.

1. Retrieve the OIDC Issuer URL

Use a data source to read the existing EKS cluster:

data "aws_eks_cluster" "prod" {
  name = "prod"
}

output "oidc_issuer_url" {
  value = data.aws_eks_cluster.prod.identity[0].oidc[0].issuer
}

Running this outputs the issuer URL.

2. Create the OIDC Provider

resource "aws_iam_openid_connect_provider" "eks_oidc" {
  url = data.aws_eks_cluster.prod.identity[0].oidc[0].issuer

  client_id_list = [
    "sts.amazonaws.com"
  ]

  thumbprint_list = [
    "9e99a48a9960b14926bb7f3b02e22da2b0ab7280"
  ]
}

url

: The cluster’s OIDC issuer. client_id_list: Usually sts.amazonaws.com. thumbprint_list: Root CA fingerprint of the OIDC server (AWS default).

3. Configure the IAM Role

resource "aws_iam_role" "ebs_csi_driver" {
  name = "eks-ebs-csi-driver"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Federated = aws_iam_openid_connect_provider.eks_oidc.arn
      }
      Action = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "${data.aws_eks_cluster.prod.identity[0].oidc[0].issuer}:sub" = "system:serviceaccount:kube-system:ebs-csi-controller-sa"
        }
      }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "ebs_csi_driver_policy" {
  role       = aws_iam_role.ebs_csi_driver.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
}

4. Apply and Verify

terraform plan -out=ftplan
terraform apply "ftplan"

After applying, list OIDC providers to confirm:

aws --profile prod iam list-open-id-connect-providers

The command should return an ARN similar to:

arn:aws:iam::140023393377:oidc-provider/oidc.eks.ap-southeast-1.amazonaws.com/id/8DE440F2238C6B959FB7C65EA52AF80B

5. Update the ServiceAccount

kubectl annotate sa ebs-csi-controller-sa -n kube-system eks.amazonaws.com/role-arn=arn:aws:iam::140023393377:role/eks-ebs-csi-driver --overwrite

Fixing the Original RocketMQ Deployment

Delete the failed PVCs:

kubectl delete pvc -n rocketmq -l app=rocketmq-broker

Restart the EBS CSI controller:

kubectl delete pod -n kube-system -l app=ebs-csi-controller

Re‑apply the RocketMQ manifests: kubectl apply -f rocketmq-broker.yaml Verify the PVCs and Pods:

kubectl get pvc -n rocketmq
kubectl get pods -n rocketmq

If the OIDC provider is correctly configured, the PVCs bind to EBS volumes and the RocketMQ broker pods run normally.

Why Terraform‑Created Clusters Lack OIDC by Default

The aws_eks_cluster resource does not automatically create an OIDC provider. You must explicitly add an aws_iam_openid_connect_provider resource and reference the cluster’s issuer URL, as shown below:

resource "aws_eks_cluster" "example" {
  name     = "example-cluster"
  role_arn = aws_iam_role.cluster.arn
  vpc_config { ... }
}

resource "aws_iam_openid_connect_provider" "example" {
  url = aws_eks_cluster.example.identity[0].oidc[0].issuer
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = ["9e99a48a9960b14926bb7f3b02e22da2b0ab7280"]
}

In the presented case, only the aws_eks_cluster was created, leaving the OIDC provider missing and breaking IRSA.

Alternative: Install EBS CSI via AWS Console

If you prefer not to manage the OIDC provider manually, you can add the Amazon EBS CSI Driver from the EKS “Add‑ons” page in the AWS Management Console. AWS will automatically create the OIDC provider and the required IAM role.

This approach is simpler, but Terraform remains the preferred method for Infrastructure‑as‑Code.

Summary

The aws_iam_openid_connect_provider resource is essential for enabling IRSA in EKS, linking Kubernetes ServiceAccounts with IAM roles. In the demonstrated scenario, the missing OIDC provider caused the EBS CSI driver to fail authentication, preventing RocketMQ from provisioning storage. Adding the provider and configuring the IAM role with Terraform resolves the issue and establishes a solid foundation for future AWS service integrations.