How to Monitor Kubernetes Clusters with Splunk Metrics Store and Heapster

Kubernetes has become the de facto container orchestration platform, and monitoring its clusters is a key operations task. This guide shows how to use Splunk’s Metrics Store together with Heapster to collect CPU, memory, network and filesystem metrics from a Kubernetes cluster.

Deployment Architecture

The solution consists of:

Heapster collecting performance data (CPU, Memory, Network, File System).

Heapster’s StatsD sink sending the data to Splunk Metrics Store.

Splunk search commands and dashboards visualizing the metrics.

Preparation

Two tasks are required:

Build the latest Heapster image and push it to a public Docker registry.

Configure a Metrics Store and a network input (UDP/TCP) in Splunk.

Use TCP for the StatsD protocol because it handles larger payloads.

mkdir myheapster
mkdir myheapster/src
export GOPATH=myheapster
cd myheapster/src
git clone https://github.com/kubernetes/heapster.git
cd heapster
make container

Modify the StatsD client to use TCP instead of UDP (see the source file statsd_client.go ).

func (client *statsdClientImpl) open() error {
    var err error
    client.conn, err = net.Dial("udp", client.host)
    if err != nil {
        glog.Errorf("Failed to open statsd client connection : %v", err)
    } else {
        glog.V(2).Infof("statsd client connection opened : %+v", client.conn)
    }
    return err
}

Two Docker images are provided:

naughtytao/heapster-amd64:v1.5.0-beta.3 (UDP)

naughtytao/heapster-amd64:v1.5.0-beta.4 (TCP)

Install and Configure Heapster

Create deployment.yaml and service.yaml and apply them with kubectl apply -f *.yaml. The sink configuration must point to the Splunk host and port, and adjust numMetricsPerMsg for UDP.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: heapster
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: heapster
        version: v6
    spec:
      containers:
      - name: heapster
        image: naughtytao/heapster-amd64:v1.5.0-beta.3
        imagePullPolicy: Always
        command:
        - /heapster
        - --source=kubernetes:https://kubernetes.default
        - --sink=statsd:udp://<splunk_ip>:<port>?numMetricsPerMsg=1

After deployment, the Heapster pod logs show successful connection to Splunk.

Monitoring in Splunk

Heapster sends metrics via StatsD to the Metrics Store. Use Splunk SPL commands to explore the data, for example: | mcatalog values(metric_name) To visualize cluster‑wide CPU usage:

| mstats avg(_value) WHERE metric_name=cluster.cpu/usage_rate span=30m

Memory usage for the kube-system namespace can be queried similarly:

| mstats avg(_value) WHERE metric_name=namespace.kube-system.memory/usage span=30m

Dashboards can be built with real‑time panels for continuous monitoring.

References

https://github.com/DataDog/the-monitor/blob/master/kubernetes/how-to-collect-and-graph-kubernetes-metrics.md

https://kubernetes.io/docs/tasks/debug-application-cluster/resource-usage-monitoring/

https://kubernetes.io/docs/tasks/debug-application-cluster/core-metrics-pipeline/

https://itnext.io/kubernetes-monitoring-with-prometheus-in-15-minutes-8e54d1de2e13

http://docs.splunk.com/Documentation/Splunk/7.0.1/Metrics/GetStarted