Rapidly Build a ClickHouse + Grafana Observability Stack – Log Collection Guide

Introduction

ClickHouse is a high‑performance OLAP database widely used for real‑time analysis of time‑series data, including logs, metrics, and traces. This guide focuses on the log pillar, showing how to collect, store, query, and visualize logs using ClickHouse, Grafana, and a modern Kubernetes environment.

Architecture Overview

Most observability agents follow a common pattern of agents (collectors) and aggregators (gateways). For small deployments a single DaemonSet‑deployed agent can send data directly to ClickHouse via HTTP. Larger deployments introduce an aggregator (or gateway) to batch data, enrich it, apply filtering, and ensure high availability. The article recommends asynchronous inserts for agents that flush frequently to avoid many small inserts.

Agents

The ClickHouse community recommends four primary agents: OpenTelemetry Collector, Vector, Fluent Bit, and Fluentd (the latter shares the same architecture as Fluent Bit). The guide demonstrates each agent using its official Helm chart, exposing the values.yaml used for configuration.

OpenTelemetry (OTEL) Collector (alpha)

OTEL Collector provides receivers, processors, and exporters. The ClickHouse exporter lives in the contrib image and supports logs and traces (metrics support is pending). The exporter uses the native ClickHouse format via the official Go client. Important considerations include:

The data model and schema are hard‑coded; users must create the target table before deployment.

The exporter is an alpha‑stage component packaged in the contrib Docker image.

Although the collector can handle >1 TB of logs, users should follow OTEL best‑practice guidance because its log support is newer than Fluent Bit or Vector.

Vector (beta)

Vector, maintained by DataDog, is a Rust‑based, vendor‑agnostic pipeline with sources → transforms → sinks . ClickHouse is supported via a beta sink that sends JSON batches over HTTP. The sink enforces a fixed data model; users must create the destination table and can use the skip_unknown_fields option to ignore columns not present in the table.

CREATE DATABASE vector;
CREATE TABLE vector.vector_logs (
   file String,
   timestamp DateTime64(3),
   kubernetes_container_id LowCardinality(String),
   kubernetes_container_image LowCardinality(String),
   kubernetes_container_name LowCardinality(String),
   kubernetes_namespace_labels Map(LowCardinality(String), String),
   kubernetes_pod_annotations Map(LowCardinality(String), String),
   kubernetes_pod_ip IPv4,
   kubernetes_pod_ips Array(IPv4),
   kubernetes_pod_labels Map(LowCardinality(String), String),
   kubernetes_pod_name LowCardinality(String),
   kubernetes_pod_namespace LowCardinality(String),
   kubernetes_pod_node_name LowCardinality(String),
   kubernetes_pod_owner LowCardinality(String),
   kubernetes_pod_uid LowCardinality(String),
   message String,
   source_type LowCardinality(String),
   stream Enum('stdout','stderr')
) ENGINE = MergeTree ORDER BY (kubernetes_container_name, timestamp);

Fluent Bit

Fluent Bit is a lightweight C‑based log forwarder. It uses generic inputs, parsers, filters, and outputs. ClickHouse support is provided via the HTTP output plugin, which writes JSONEachRow rows. Because the plugin does not batch, users should configure a larger flush interval (e.g., ≥10 s) and enable asynchronous inserts with wait_for_async_insert (0 for best throughput, 1 for stronger delivery guarantees). The article also shows how to move Kubernetes metadata from a nested kubernetes column into top‑level Map columns to avoid column explosion.

CREATE TABLE fluent.fluent_logs (
    timestamp DateTime64(9),
    `log` String,
    kubernetes Map(LowCardinality(String), String),
    host LowCardinality(String),
    pod_name LowCardinality(String),
    stream LowCardinality(String),
    labels Map(LowCardinality(String), String),
    annotations Map(LowCardinality(String), String)
) ENGINE = MergeTree ORDER BY (host, pod_name, timestamp);

Kubernetes Deployment

For log‑only scenarios the simplest deployment uses the official Helm charts for each agent. The article links to the full Helm values for agents, aggregators, and the ClickHouse Cloud trial cluster. It also notes that the ClickHouse TTL feature can be used to drop old partitions, and that the example sets TTL to 0 (disabled) for demonstration.

Data Model & Schema

Three example tables are created: otel.otel_logs – used by the OTEL Collector. vector.vector_logs – used by Vector. fluent.fluent_logs – used by Fluent Bit.

Each table defines a Map(LowCardinality(String), String) column for Kubernetes labels/annotations and several secondary indexes (bloom filters) for fast look‑ups.

SHOW CREATE TABLE otel.otel_logs;
CREATE TABLE otel.otel_logs (
    Timestamp DateTime64(9) CODEC(Delta(8), ZSTD(1)),
    TraceId String CODEC(ZSTD(1)),
    SpanId String CODEC(ZSTD(1)),
    TraceFlags UInt32 CODEC(ZSTD(1)),
    SeverityText LowCardinality(String) CODEC(ZSTD(1)),
    SeverityNumber Int32 CODEC(ZSTD(1)),
    ServiceName LowCardinality(String) CODEC(ZSTD(1)),
    Body String CODEC(ZSTD(1)),
    ResourceAttributes Map(LowCardinality(String), String) CODEC(ZSTD(1)),
    LogAttributes Map(LowCardinality(String), String) CODEC(ZSTD(1)),
    INDEX idx_trace_id TraceId TYPE bloom_filter(0.001) GRANULARITY 1,
    INDEX idx_res_attr_key mapKeys(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_res_attr_value mapValues(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_log_attr_key mapKeys(LogAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_log_attr_value mapValues(LogAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_body Body TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 1
) ENGINE = MergeTree PARTITION BY toDate(Timestamp) ORDER BY (ServiceName, SeverityText, toUnixTimestamp(Timestamp), TraceId) SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1;

Compression Results

ClickHouse’s columnar storage yields high compression. The article runs a query across the three example databases and reports ratios ranging from 14× to 33×, with label/annotation columns achieving >200× compression because they are sparse.

SELECT database, table,
       formatReadableSize(sum(data_compressed_bytes)) AS compressed_size,
       formatReadableSize(sum(data_uncompressed_bytes)) AS uncompressed_size,
       round(sum(data_uncompressed_bytes)/sum(data_compressed_bytes),2) AS ratio
FROM system.columns
WHERE database IN ('fluent','vector','otel')
  AND name NOT LIKE '%labels%'
  AND name NOT LIKE '%annotations%'
GROUP BY database, table
ORDER BY database, table;

Typical Queries

Because logs are time‑series data, most dashboards aggregate by time and then filter by pod, namespace, or error pattern. Example queries include:

Log count per pod per day with WITH FILL to fill missing days.

Log entries for a specific pod within a time window.

Extracting values from Map columns, e.g., kubernetes_pod_labels['statefulset.kubernetes.io/pod-name'].

Finding containers whose messages contain a substring or match a regular expression.

SELECT toStartOfInterval(timestamp, toIntervalDay(1)) AS time,
       kubernetes_pod_name AS pod_name,
       count() AS c
FROM vector.vector_logs
GROUP BY time, pod_name
ORDER BY pod_name ASC, time ASC WITH FILL STEP toIntervalDay(1)
LIMIT 5;

Query Performance Tuning

The primary factor for query speed is the ORDER BY key defined at table creation. Columns frequently used in WHERE clauses should appear early in the key, ordered by increasing cardinality (e.g., host → container → timestamp). Keys longer than three or four columns rarely provide additional benefit. For Map columns, accessing a sub‑key loads the whole map; if a particular label is queried often, consider extracting it to a dedicated column.

Visualization

The recommended visualization tool is the official ClickHouse datasource plugin for Grafana. The article points to a pre‑built dashboard (ID 17284) that visualizes Kubernetes logs collected via Fluent Bit. Users can import the dashboard JSON or use the snapshot link provided.

Conclusion

The guide demonstrates that a variety of agents and pipelines can reliably collect and store logs in ClickHouse, whether running on Kubernetes or on traditional servers. It also outlines next steps: exploring back‑pressure, delivery guarantees, adding metrics and traces, and further schema optimizations.