Building a Cloud‑Native ClickHouse PaaS to Power Sentry Snuba Event Analytics

Design Goals

The aim is to create a high‑performance, low‑cost, cloud‑native ClickHouse platform that supports multi‑cloud orchestration, automated operations, elastic scaling, self‑healing, tenant isolation, RBAC, and audit logging, and to use it as the storage backend for Sentry's Snuba event‑analysis engine.

Architecture Overview

Snuba builds on ClickHouse tables and materialized views, ingesting data from Kafka streams and exposing both point‑in‑time and streaming queries. The official Altinity.Cloud ClickHouse PaaS provides a reference multi‑tenant architecture.

Kubernetes Operator Basics

A Kubernetes Operator packages the lifecycle management of an application. The article introduces the Altinity ClickHouse Operator (the industry‑leading open‑source provider) and the RadonDB ClickHouse Operator, both of which can create and manage ClickHouse clusters via CRDs.

Experimental Environment

VKE K8s cluster (v1.23.14) on Vultr

Kubesphere v3.3.1 for visual management

Longhorn 1.14 for distributed block storage

Deploying the ClickHouse Operator

# values.operator.yaml
watchAllNamespaces: true
enablePrometheusMonitor: true

Install the operator in the kube-system namespace:

cd vip-k8s-paas/10-cloud-native-clickhouse
helm install clickhouse-operator ./clickhouse-operator -f values.operator.yaml -n kube-system
kubectl -n kube-system get po | grep clickhouse-operator

Deploying a ClickHouse Cluster

# values.cluster.yaml (excerpt)
clickhouse:
  clusterName: snuba-clickhouse-nodes
  shardscount: 2
  replicascount: 2
zookeeper:
  install: true
  replicas: 3

Install the cluster:

kubectl create ns cloud-clickhouse
helm install clickhouse ./clickhouse-cluster -f values.cluster.yaml -n cloud-clickhouse
kubectl get po -n cloud-clickhouse

Scaling Shards with the Operator

Edit the Chi resource to change shardsCount from 2 to 3: kubectl edit chi/clickhouse -n cloud-clickhouse After saving, new pods chi-clickhouse-snuba-ck-nodes-2-0-0 and chi-clickhouse-snuba-ck-nodes-2-1-0 appear, showing the operator automatically creates the extra shard and replicas.

Creating Distributed Tables

Connect to each pod with clickhouse-client and run:

create database test on cluster 'snuba-ck-nodes';
CREATE TABLE test.t_local on cluster 'snuba-ck-nodes' (
    EventDate DateTime,
    CounterID UInt32,
    UserID UInt32
) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/test/t_local', '{replica}')
PARTITION BY toYYYYMM(EventDate)
ORDER BY (CounterID, EventDate, intHash32(UserID))
SAMPLE BY intHash32(UserID);

CREATE TABLE test.t_dist on cluster 'snuba-ck-nodes' (
    EventDate DateTime,
    CounterID UInt32,
    UserID UInt32
) ENGINE = Distributed('snuba-ck-nodes', test, t_local, rand());

INSERT INTO test.t_dist VALUES ('2022-12-16 00:00:00',1,1),('2023-01-01 00:00:00',2,2),('2023-02-01 00:00:00',3,3);
SELECT * FROM test.t_dist;

Analyzing the Original Sentry Helm Charts

The upstream sentry‑kubernetes/charts bundles ClickHouse and Zookeeper as dependencies, which couples all middleware together and hinders independent scaling. The article extracts the Chart.yaml showing these dependencies and proposes disabling the built‑in ClickHouse and Zookeeper.

Deploying Independent Zookeeper

# values.yaml (excerpt)
metrics:
  containerPort: 9141
  enabled: true
service:
  type: ClusterIP
  headless:
    publishNotReadyAddresses: true

Install:

kubectl create ns cloud-zookeeper-paas
helm install zookeeper ./zookeeper -f values.yaml -n cloud-zookeeper-paas

Verify the service address zookeeper.cloud-zookeeper-paas.svc.cluster.local:2181 and inspect zoo.cfg inside the pod.

Deploying Independent ClickHouse

Use the Sentry‑maintained ClickHouse chart with a compatible image ( yandex/clickhouse-server:20.8.19.4) and customize the Zookeeper server list in values.yaml:

clickhouse:
  configmap:
    zookeeper_servers:
      config:
        - hostTemplate: 'zookeeper-0.zookeeper-headless.cloud-zookeeper-paas.svc.cluster.local'
          index: clickhouse
          port: "2181"
        - hostTemplate: 'zookeeper-1.zookeeper-headless.cloud-zookeeper-paas.svc.cluster.local'
          index: clickhouse
          port: "2181"
        - hostTemplate: 'zookeeper-2.zookeeper-headless.cloud-zookeeper-paas.svc.cluster.local'
          index: clickhouse
          port: "2181"
      enabled: true
metrics:
  enabled: true

Install:

kubectl create ns cloud-clickhouse-paas
helm install clickhouse ./clickhouse -f values.yaml -n cloud-clickhouse-paas

Check ConfigMaps clickhouse-config, clickhouse-metrica, and clickhouse-users for the generated config.xml, metrica.xml, and users.xml files.

Customizing Sentry to Use the External ClickHouse PaaS

# values.yaml (excerpt)
clickhouse:
  enabled: false
zookeeper:
  enabled: false
externalClickhouse:
  database: default
  host: "clickhouse.cloud-clickhouse-paas.svc.cluster.local"
  httpPort: 8123
  tcpPort: 9000
  username: default
  password: ""
  singleNode: false
  clusterName: "clickhouse"

Deploy Sentry:

helm install sentry ./sentry -f values.yaml -n sentry

After deployment, verify that the _local and _dist tables exist and that system.zookeeper contains the expected paths.

Advanced Scaling & High‑Throughput Scenarios

The article outlines how to combine multiple VKE clusters, separate Zookeeper‑Operator and ClickHouse‑Operator instances, and distribute schemas across many clusters, nodes, shards, and replicas to achieve massive throughput. Detailed source‑code analysis and test‑case inspection are left for future live sessions.