Test Your Grafana Knowledge: 8 Interview Questions with Answers

Core Grafana concepts and relationships

Grafana revolves around five core objects:

DataSource – connection to a backend such as Prometheus, MySQL, Loki, etc.

Dashboard – a collection of Panels.

Panel – a single visualization (graph, stat, table, …).

Variable – a templated dropdown that can be referenced in queries with $var.

Annotation – time‑axis markers (e.g., releases, incidents).

Relationship flow:

An Organization contains DataSources and Dashboards.

A Dashboard contains Panels.

Each Panel references a DataSource.

Panel queries may reference Variables.

Query Variables pull their option list from a DataSource.

Annotations are rendered on the Panel time axis.

# DataSource: prometheus-huadong
# Variable: $service (query job label from prometheus-huadong)
# Panel: QPS line chart
#   datasource: prometheus-huadong
#   query: rate(http_requests_total{job="$service"}[5m])
# Annotation: release events
#   datasource: prometheus-huadong
#   query: deploy_events{action="release"}

Dashboard design principles (single responsibility)

Single responsibility – a Dashboard answers one type of question.

Variable‑driven – use Variables instead of duplicating Dashboards.

Threshold coloring – highlight anomalies with color rules.

Alert overlay – show alert status directly on Panels.

Information hierarchy – layout progresses top‑to‑bottom, left‑to‑right.

Unified time range – a global time picker dominates.

Typical folder organization separates service overviews, business metrics and infrastructure monitoring, avoiding a “catch‑all” Dashboard.

业务监控/
├── 订单服务总览/
├── 支付服务总览/
└── 用户服务总览/
基础设施/
├── 主机监控/
├── 数据库监控/
└── 网络监控/

Panel types and when to use them

Time Series – trend over time (QPS, latency, CPU).

Stat – single KPI (online users, today orders).

Gauge – single value gauge (CPU, memory).

Bar Gauge – horizontal multi‑value comparison (region QPS, error rates).

Heatmap – distribution density (latency distribution, request hot spots).

Node Graph – topology (service call graph, network).

Geomap – geographical distribution (multi‑region devices, CDN nodes).

Logs – log streams (application, error logs).

Table – tabular data (instance list, alerts).

Pie Chart – proportion (status code distribution).

Bar Gauge vs Time Series – use Bar Gauge when multiple series overlap at a single point in time (e.g., four regions' QPS) or when only the current value matters; use Time Series for temporal trends.

# Wrong: 4‑region QPS as Time Series (lines tangled)
type: timeseries
targets:
  - expr: 'sum by (region) (rate(http_requests_total[5m]))'

# Correct: Bar Gauge horizontal comparison
type: bargauge
orientation: 'horizontal'
options:
  displayMode: 'gradient'

Heatmap solves distribution‑density visualization. A traditional P95 line shows only a single value; a heatmap reveals single‑peak vs multi‑peak and long‑tail behavior.

type: heatmap
targets:
  - expr: 'sum by (le) (rate(http_request_duration_seconds_bucket[5m]))'
fieldConfig:
  defaults:
    unit: 's'

Variables – templating, types, and chain implementation

Variables eliminate Dashboard duplication. Example: a 3‑environment × 4‑service matrix would require 12 Dashboards; with Variables a single Dashboard plus three Variables (env, service, instance) suffices.

Query Variable – options fetched from a DataSource (e.g., label_values(http_requests_total, job)).

Custom Variable – manually defined list (e.g., prod,staging,dev).

Interval Variable – time intervals (1m,5m,10m,1h).

Data Source Variable – switch data sources.

Text Box Variable – free‑form input.

Constant Variable – shared constant.

Chain Variables example (region → env → service): each selection triggers the next query.

# Variable 1: $region
name: region
type: query
datasource: prometheus
query: label_values(up, region)
refresh: 1
includeAll: true

# Variable 2: $env (depends on $region)
name: env
type: query
datasource: prometheus
query: label_values(up{region=~"$region"}, env)
refresh: 1
includeAll: true

# Variable 3: $service (depends on $region & $env)
name: service
type: query
datasource: prometheus
query: label_values(up{region=~"$region", env=~"$env"}, job)
refresh: 1
includeAll: true

Panel query using the variables:

sum by (instance) (rate(http_requests_total{job="$service", env="$env", region="$region"}[$interval]))

Grafana Alerting vs Alertmanager – complementary relationship

Both systems can coexist; they differ in data‑source support, rule storage, routing, maturity and multi‑source capabilities.

Data source : Grafana Alerting works with any Grafana‑supported source; Alertmanager only with Prometheus.

Rule location : Grafana stores rules in its DB or provisioning YAML; Alertmanager uses Prometheus rule files.

Routing : Grafana uses built‑in Contact Points; Alertmanager is a standalone component.

Multi‑source alerts : strong in Grafana, weak in Alertmanager.

Ecosystem maturity : Grafana Alerting (Grafana 8+), Alertmanager (CNCF‑graduated).

Recommended production setup:

Prometheus alerts → Alertmanager (single‑source, mature governance).

Cross‑source alerts → Grafana Alerting.

Run both, ensuring no duplicate alerts (disable manageAlerts on the Prometheus DataSource in Grafana).

# Alertmanager rule (Prometheus alerts)
groups:
  - name: service-alerts
    rules:
      - alert: ServiceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Service {{ $labels.job }} down"

# Grafana Alerting rule (cross‑source)
groups:
  - name: business-alerts
    interval: 30s
    rules:
      - uid: 'mysql-slow-query'
        title: 'MySQL 慢查询过多'
        condition: A
        data:
          - refId: A
            datasourceUid: 'mysql-business'
            model:
              rawSql: "SELECT COUNT(*) FROM slow_log WHERE duration > 5"
        for: 5m
        labels:
          severity: warning
        notificationSettings:
          contact_point: 'dingtalk-ops'

Provisioning – configuration‑as‑code

Provisioning moves Dashboard and DataSource definitions from the UI into version‑controlled files, enabling CI/CD sync across multiple Grafana instances.

grafana/provisioning/
├── datasources/
│   └── datasources.yaml
├── dashboards/
│   ├── dashboards.yaml
│   └── projects/
│       ├── gateway-overview.json
│       └── service-overview.json
└── alerting/
    ├── contactpoints.yaml
    ├── policies.yaml
    └── rules.yaml

DataSource YAML example (Prometheus, Loki, MySQL):

# provisioning/datasources/datasources.yaml
apiVersion: 1
datasources:
  - name: prometheus-huadong
    type: prometheus
    access: proxy
    url: http://prometheus-huadong:9090
    isDefault: true
    jsonData:
      timeInterval: 15s
      httpMethod: POST
  - name: prometheus-huanan
    type: prometheus
    access: proxy
    url: http://prometheus-huanan:9090
    jsonData:
      timeInterval: 15s
  - name: loki
    type: loki
    access: proxy
    url: http://loki:3100
  - name: mysql-business
    type: mysql
    access: proxy
    url: mysql:3306
    database: business
    user: readonly
    secureJsonData:
      password: ReadOnly@123
    jsonData:
      sslmode: disable

Dashboard provisioning YAML (loads JSON files from a folder):

# provisioning/dashboards/dashboards.yaml
apiVersion: 1
providers:
  - name: 'business-dashboards'
    orgId: 1
    folder: '业务监控'
    folderUid: 'business'
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    allowUiUpdates: true
    options:
      path: /etc/grafana/provisioning/dashboards/projects
      foldersFromFilesStructure: true

Alerting provisioning files (contact points, policies, rules) follow the same pattern. Setting allowUiUpdates: false forces all changes through Git; true permits UI edits to be written back.

Typical CI/CD flow:

Edit Dashboard JSON, open a PR.

Merge after review.

CI pushes the updated files to the Grafana server.

Grafana reloads automatically (controlled by updateIntervalSeconds).

High availability (HA) for Grafana

HA requires three components:

Multiple stateless Grafana instances (e.g., behind an Nginx load balancer).

A shared relational database (MySQL or PostgreSQL) instead of the default SQLite.

Shared session storage (Redis).

Why SQLite is unsuitable for production :

File‑level write lock – whole‑file locked on every write, poor concurrency.

No support for multiple instances – file lock conflicts.

No replication – single point of failure.

MySQL configuration (grafana.ini) :

# grafana.ini
[database]
type = mysql
host = mysql:3306
name = grafana
user = grafana
password = Grafana@123
conn_max_idle_time = 2
conn_max_lifetime = 14400
max_idle_conn = 50
max_open_conn = 100

Session sharing with Redis :

# grafana.ini
[session]
provider = redis
provider_config = addr=redis:6379,pool_size=100,db=grafana
cookie_name = grafana_sess
cookie_secure = true

Docker‑compose example (two Grafana nodes, MySQL, Redis, Nginx LB) :

services:
  grafana-1:
    image: grafana/grafana:latest
    environment:
      - GF_DATABASE_TYPE=mysql
      - GF_DATABASE_HOST=mysql:3306
      - GF_DATABASE_NAME=grafana
      - GF_DATABASE_USER=grafana
      - GF_DATABASE_PASSWORD=Grafana@123
      - GF_SESSION_PROVIDER=redis
      - GF_SESSION_PROVIDER_CONFIG=addr=redis:6379,pool_size=100,db=grafana
    depends_on:
      - mysql
      - redis

  grafana-2:
    image: grafana/grafana:latest
    environment: *same_as_grafana-1
    depends_on:
      - mysql
      - redis

  mysql:
    image: mysql:5.7
    environment:
      - MYSQL_DATABASE=grafana
      - MYSQL_USER=grafana
      - MYSQL_PASSWORD=Grafana@123
      - MYSQL_ROOT_PASSWORD=RootPass

  redis:
    image: redis:6

  nginx:
    image: nginx:latest
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
    # load‑balance to grafana-1 and grafana-2

Performance tweaks (list of common knobs): query_timeout: 30s – avoid slow queries blocking the UI.

Enable dashboard caching ( enabled = true) to store query results.

Limit Panels per Dashboard (recommended ≤ 30) to keep rendering fast.

Configure DataSource connection pool ( max_open_conn) to prevent exhaustion.

Enable gzip compression ( [server].enable_gzip = true).

Design of a multi‑region multi‑service monitoring screen

Variables (chain) – region → env → service → interval:

# $region
name: region
type: query
datasource: prometheus
query: label_values(up, region)
refresh: 1
includeAll: true

# $env (depends on $region)
name: env
type: query
datasource: prometheus
query: label_values(up{region=~"$region"}, env)
refresh: 1
includeAll: true

# $service (depends on $region & $env)
name: service
type: query
datasource: prometheus
query: label_values(up{region=~"$region", env=~"$env"}, job)
refresh: 1
includeAll: true

# $interval
name: interval
type: interval
query: '1m,5m,10m,30m,1h'

Panel layout (top → middle → bottom → footer) :

Top – four Stat panels showing total QPS, error rate, P95 latency, instance count.

sum(rate(http_requests_total{region=~"$region", env=~"$env", job=~"$service"}[$interval]))
sum(rate(http_requests_total{region=~"$region", status=~"5.."}[$interval])) / sum(rate(http_requests_total{region=~"$region"}[$interval]))
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{region=~"$region", job=~"$service"}[$interval])) by (le))
count(up{region=~"$region", job=~"$service"})

Middle – Geomap visualising service health per region. sum by (region) (up{job=~"$service"}) Bottom left – Time Series for QPS trend.

sum by (region, job) (rate(http_requests_total{region=~"$region", job=~"$service"}[$interval]))

Bottom middle – Time Series for P95 latency trend.

histogram_quantile(0.95, sum by (le, region) (rate(http_request_duration_seconds_bucket{region=~"$region", job=~"$service"}[$interval])))

Bottom right – Bar Gauge comparing QPS across regions.

sum by (region) (rate(http_requests_total{job=~"$service"}[$interval]))

Bottom far‑right – Heatmap for latency distribution.

sum by (le) (rate(http_request_duration_seconds_bucket{job=~"$service"}[$interval]))

Footer – Logs panel streaming error logs. {app=~"$service"} Threshold coloring (example rules):

QPS: < 5000 green, 5000‑10000 yellow, > 10000 red.

Error rate: < 1 % green, 1‑5 % yellow, > 5 % red.

P95 latency: < 500 ms green, 500 ms‑1 s yellow, > 1 s red.

Deployment – the Dashboard JSON is placed under

grafana/provisioning/dashboards/projects/multi-region-overview.json

, the DataSource definition under provisioning/datasources/datasources.yaml. The following snippet shows the Grafana service in docker-compose.yml:

grafana:
  image: grafana/grafana:latest
  container_name: dev-grafana
  ports:
    - "3100:3000"
  environment:
    - GF_SECURITY_ADMIN_USER=grafana
    - GF_SECURITY_ADMIN_PASSWORD=Grafana@123
    - GF_INSTALL_PLUGINS=grafana-clock-panel,grafana-piechart-panel
  volumes:
    - ./data/grafana:/var/lib/grafana
    - ./grafana/provisioning:/etc/grafana/provisioning
  depends_on:
    - prometheus

After docker compose up, the screen is reachable at http://localhost:3100 using the credentials above. Changing the $region variable refreshes the entire view.