How to Monitor Business Metrics with Prometheus in Kubernetes

Prometheus Monitoring Business Metrics

With Kubernetes becoming the de‑facto standard for container orchestration, deploying micro‑services is easy, but scaling them introduces governance challenges, prompting the need for observability.

Observability combines logging, metrics, and tracing—the three pillars that help detect, locate, and prevent failures in distributed systems.

Observability Pillars

The three pillars are:

Logging records events and runtime details, offering deep insight but consuming significant storage and query resources, often mitigated with filters.

Metrics are aggregated numeric values that require little storage and reveal system state and trends; multidimensional structures like contour metrics enhance detail.

Tracing follows request flows to pinpoint anomalies, but like logging it can be resource‑intensive and usually relies on sampling.

The focus of this article is on the metrics pillar, specifically how to monitor business‑level metrics with Prometheus in a Kubernetes‑based cloud‑native stack.

Metric Definitions and Types

Metric Syntax

Metrics are expressed as:

<metric_name>{<label_name>=<label_value>, ...}

Metric names may contain ASCII letters, digits, underscores, and colons and must match [a-zA-Z_:][a-zA-Z0-9_:]*. Labels are limited to ASCII letters, digits, and underscores ( [a-zA-Z_][a-zA-Z0-9_]*).

Metric Types

Prometheus defines four metric types:

Counter – monotonically increasing values (e.g., http_requests_total); usually suffixed with _total.

Gauge – values that can go up or down (e.g., node_memory_MemFree).

Summary – provides quantiles and aggregates for latency‑type data.

Histogram – similar to Summary but exposes bucket counts ( _count) and sums ( _sum) and supports histogram_quantile().

Application Metric Monitoring

Exposing Metrics

Prometheus pulls metrics via HTTP. Services should expose a /metrics endpoint that the Prometheus server scrapes.

Defining Metrics (Go example)

var (
    // HTTP request duration (Histogram)
    HTTPReqDuration *prometheus.HistogramVec
    // HTTP request total (Counter)
    HTTPReqTotal *prometheus.CounterVec
    // Task running count (Gauge)
    TaskRunning *prometheus.GaugeVec
)

func init() {
    HTTPReqDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
        Name: "http_request_duration_seconds",
        Help: "http request latencies in seconds",
    }, []string{"method", "path"})

    HTTPReqTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "total number of http requests",
    }, []string{"method", "path", "status"})

    TaskRunning = prometheus.NewGaugeVec(prometheus.GaugeOpts{
        Name: "task_running",
        Help: "current count of running task",
    }, []string{"type", "state"})

    prometheus.MustRegister(HTTPReqDuration, HTTPReqTotal, TaskRunning)
}

Generating Metrics (middleware example)

start := time.Now()
// ... request handling ...
duration := float64(time.Since(start)) / float64(time.Second)
path := c.Request.URL.Path

controllers.HTTPReqTotal.With(prometheus.Labels{"method": c.Request.Method, "path": path, "status": strconv.Itoa(c.Writer.Status())}).Inc()
controllers.HTTPReqDuration.With(prometheus.Labels{"method": c.Request.Method, "path": path}).Observe(duration)
controllers.TaskRunning.With(prometheus.Labels{"type": shuffle([]string{"video", "audio"}), "state": shuffle([]string{"process", "queue"})}).Inc()
// on task completion
controllers.TaskRunning.With(prometheus.Labels{"type": shuffle([]string{"video", "audio"}), "state": shuffle([]string{"process", "queue"})}).Dec()

Scraping Configuration

scrape_interval: 5s
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['prometheus:9090']
  - job_name: 'local-service'
    metrics_path: /metrics
    static_configs:
      - targets: ['host.docker.internal:8000']

In production Kubernetes clusters, static target lists are replaced by service discovery modes (Node, Service, Pod, Endpoints, Ingress) integrated with the Kubernetes API.

Metric dashboards can be visualized in Grafana, as shown in the accompanying screenshots.