How to Leverage Kubernetes Audit Logs for Secure Monitoring and Analysis

Kubernetes Audit Log Overview

Kubernetes has offered audit logging since version 1.7, providing time‑stamped records of API requests and responses, including user, resource, verb, and outcome. Audit logs enable administrators to trace who performed which actions, when, and from where, which is essential for security monitoring and post‑incident analysis.

Log Format and Policy

Audit entries are standard JSON objects written by the API server according to a configurable audit policy. The policy can define retention time, rotation, and which resources are logged. Detailed documentation is available in the official Kubernetes audit guide.

Recording Stages

RequestReceived

– logged when the request is received before a handler is chosen. ResponseStarted – logged after response headers are sent but before the body, useful for long‑running operations such as watches. ResponseComplete – logged after the response is fully sent. Panic – logged when the API server panics.

Recording Levels

None

– no audit data is recorded. Metadata – records only request metadata (user, timestamp, verb, resource) without request or response bodies. Request – records metadata and request body. RequestResponse – records full metadata, request body, and response body.

Audit Log Analysis

Most vendors currently stop at collecting audit logs into a log store and providing keyword search. A Metadata‑level entry already contains more than 20 fields; Request or RequestResponse levels can contain hundreds, making manual analysis costly.

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1beta1",
  "metadata": {"creationTimestamp": "2019-01-14T07:48:38Z"},
  "level": "Metadata",
  "timestamp": "2019-01-14T07:48:38Z",
  "auditID": "cf2915c0-0b43-4e1d-9d66-fbae481a0e0a",
  "stage": "ResponseComplete",
  "requestURI": "/apis/authentication.k8s.io/v1beta1?timeout=32s",
  "verb": "get",
  "user": {
    "username": "system:serviceaccount:kube-system:generic-garbage-collector",
    "uid": "cd3fbe04-0508-11e9-965f-00163e0c7cbe",
    "groups": ["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]
  },
  "sourceIPs": ["192.168.0.249"],
  "responseStatus": {"metadata": {}, "code": 200},
  "requestReceivedTimestamp": "2019-01-14T07:48:38.214979Z",
  "stageTimestamp": "2019-01-14T07:48:38.215102Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"system:discovery\" of ClusterRole \"system:discovery\" to Group \"system:authenticated\""
  }
}

Alibaba Cloud Audit Log Solution

Alibaba Cloud Container Service enables API server audit logging by default and forwards the logs to Log Service (SLS). SLS automatically creates indexes, dashboards, and supports real‑time queries. Administrators can view audit reports directly in the console or configure custom alerts and analyses in SLS.

Enable APIServer audit logging with a secure, generic policy.

Logtail collects the audit logs and pushes them to SLS.

SLS creates indexes and visual reports for the logs.

The console displays audit dashboards and metrics.

Users can add alerts or custom queries in SLS as needed.

Audit Reports

Three default reports are provided:

Audit Center Overview – high‑level event summary and details of critical events such as public access or secret reads.

Resource Operation Overview – statistics for compute, network, and storage resources (e.g., Deployments, Services, ConfigMaps).

Resource Detailed Operation List – per‑resource event list with timestamps, namespaces, success rates, and raw log links.

All reports support filtering by time range, sub‑account ID, namespace, etc., enabling quick identification of resource changes, trend patterns, responsible accounts, source IP characteristics, and detailed event data.

Custom Alerts

Users can create alerts on specific audit events, such as public‑access attempts, in three steps:

In the audit report, click the advanced options and choose “Create Alert”.

Enter alert name, target event, and condition.

Configure notification method and content.

Custom Analysis

If the built‑in dashboards do not meet requirements, SLS SQL, dashboards, and other features can be used for bespoke analysis and visualization.