How to Enable and Ship Kubernetes Audit Logs to Elasticsearch with Filebeat and Logstash

Kubernetes Auditing Overview

Kubernetes auditing provides a chronological, security‑related record of every user, application, and control‑plane activity performed via the Kubernetes API.

1. Configure the API Server for Auditing

Edit the kube‑apiserver manifest ( /etc/kubernetes/manifests/kube-apiserver.yaml) to add the audit flags and mount the audit log directory.

# Modify apiserver startup parameters
- --audit-policy-file=/etc/kubernetes/pki/audit-policy.yaml
- --audit-log-path=/var/log/kubernetes/audit.json
- --audit-log-maxage=3
- --audit-log-format=json

# Mount audit directory
volumeMounts:
- mountPath: /var/log/kubernetes
  name: audit-log

volumes:
- hostPath:
    path: /var/log/kubernetes
    type: DirectoryOrCreate
  name: audit-log

Tip: Edit /etc/kubernetes/manifests/kube-apiserver.yaml directly.

2. Define the Audit Policy

Create /etc/kubernetes/pki/audit-policy.yaml with a policy that records metadata for all resources.

apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
- level: Metadata   # Record request metadata only, no request/response bodies

Tip: Save the above content to /etc/kubernetes/pki/audit-policy.yaml.

3. Restart the API Server

Move the manifest out of the way and back to trigger a restart.

sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp
sudo mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml

4. Collect Audit Logs with Filebeat

Reuse the existing Filebeat Helm deployment and add an input that reads /var/log/kubernetes/audit.json.

- type: log
  enabled: true
  fields:
    log_topic: k8s-audit
  paths:
    - /var/log/kubernetes/audit.json

Tip: Add the configuration under daemonset.filebeatConfig.filebeat.inputs.

helm -n obs-system upgrade filebeat -f filebeat-values.yaml filebeat

5. Forward Audit Logs to Elasticsearch with Logstash

Define a Logstash pipeline that parses the JSON audit records and outputs them to an index template k8s-audit in Elasticsearch.

PUT _index_template/k8s-audit
{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "jiaxzeng",
          "rollover_alias": "k8s-audit"
        },
        "number_of_shards": "3",
        "number_of_replicas": "1"
      }
    }
  },
  "index_patterns": ["k8s-audit*"]
}

Logstash pipeline (k8s-audit.conf):

input {
  kafka {
    bootstrap_servers => "172.139.20.17:9095,172.139.20.81:9095,172.139.20.177:9095"
    topics => ["k8s-audit"]
    group_id => "k8s-audit"
    security_protocol => "SASL_SSL"
    sasl_mechanism => "SCRAM-SHA-512"
    sasl_jaas_config => "org.apache.kafka.common.security.scram.ScramLoginModule required username='admin' password='admin-password';"
    ssl_truststore_location => "/usr/share/logstash/certs/kafka/kafka.server.truststore.p12"
    ssl_truststore_password => "truststore_password"
    ssl_truststore_type => "PKCS12"
  }
}
filter {
  json { source => "message" }
  mutate { remove_field => ["@timestamp","@metadata","log","fields","input","ecs","agent"] }
}
output {
  elasticsearch {
    hosts => ["https://elasticsearch.obs-system.svc:9200"]
    ilm_enabled => true
    ilm_rollover_alias => "k8s-audit"
    ilm_pattern => "{now/d}-000001"
    ilm_policy => "jiaxzeng"
    manage_template => false
    template_name => "k8s-audit"
    user => "elastic"
    password => "admin@123"
    ssl => true
    ssl_certificate_verification => true
    truststore => "/usr/share/logstash/certs/es/http.p12"
    truststore_password => "http.p12"
  }
}

Tip: Add the above configuration to the logstashPipeline section of your Helm values.

helm -n obs-system upgrade logstash -f logstash-values.yaml logstash

6. Visualize Audit Logs in Kibana

Create an index pattern k8s-audit* in Kibana’s Stack Management, then explore the data via the Discover tab.