Detect and Alert Dangerous kubectl exec/port-forward in EKS with CloudWatch

Introduction

Kubectl is a powerful tool for interacting with a Kubernetes cluster, but the kubectl exec and kubectl port-forward sub‑commands can become serious security backdoors if left unchecked. In production Amazon EKS clusters, monitoring these operations is essential to prevent credential theft, lateral movement, or malicious code injection.

Why Audit Logs Matter

AWS EKS provides Control Plane Logging, and the Audit Logs component records every request to the Kubernetes API server in structured JSON. By enabling this log type, you can capture the user, verb, and objectRef for each operation, including the high‑risk exec and port‑forward calls.

Step‑by‑Step Implementation

Enable Audit Logging Run the following AWS CLI command, replacing your-cluster-name with your actual cluster name, to turn on audit logs.

# Replace "your-cluster-name" with your EKS cluster name
aws eks update-cluster-config \
    --name your-cluster-name \
    --logging '{"clusterLogging":[{"types":["audit"],"enabled":true}]}'

After a few minutes, EKS creates a CloudWatch log group named /aws/eks/your-cluster-name/cluster where all audit entries are streamed.

Analyze Logs and Create Metric Filter Execute a test kubectl exec command to generate a log entry, then locate the JSON record in CloudWatch. The relevant fields look like: "verb": "create" – the action type for both exec and port‑forward. "objectRef": { "subresource": "exec" } or { "subresource": "portforward" } – identifies the risky sub‑resource.

Create a metric filter that matches these patterns:

{ ($.verb = "create") && (($.objectRef.subresource = "exec") || ($.objectRef.subresource = "portforward")) }

Assign the filter a name such as HighRiskKubectlOperations , a custom namespace EKS/Audit , and a metric name HighRiskOperationCount with a value of 1 for each match.

Set Up Real‑Time Alarm In the CloudWatch console, create an alarm on the metric HighRiskOperationCount with the following conditions:

Statistic: Sum

Period: 1 minute

Threshold: > 0

Configure the alarm to send a notification via an SNS topic (email, SMS, Lambda, Slack, etc.). When the alarm state changes to ALARM , you receive an alert containing the timestamp, user, and pod details.

Further Enhancements

Fine‑Grained Filters – Add conditions on user.username or objectRef.namespace to ignore trusted users or limit alerts to production namespaces.

Log Archiving & Analysis – Export CloudWatch logs to S3 and use Amazon Athena or OpenSearch for long‑term analytics and visualizations.

Automated Response – Subscribe a Lambda function to the SNS topic to automatically isolate the offending pod or post a detailed incident report to a security channel.

Conclusion

By leveraging EKS audit logs, CloudWatch metric filters, and SNS alarms, you can turn the Kubernetes API server into a “watchdog” that instantly flags dangerous kubectl exec and kubectl port-forward actions, dramatically improving cluster security and compliance.