Understanding Snuba: Kafka‑Powered ClickHouse Data Platform for Sentry

Snuba

is a service built on ClickHouse that provides rich data models, fast ingestion directly from Kafka, and a rule‑based query optimizer for Sentry error aggregation.

Features

Database access layer for ClickHouse distributed storage.

Graph‑logical data model queried via SnQL, a SQL‑like language.

Supports multiple independent datasets in a single installation.

Rule‑based query optimizer.

Migration system applying DDL changes to ClickHouse in both single‑node and distributed environments.

Direct data ingestion from Kafka.

Supports point‑in‑time and streaming queries.

Use Cases in Sentry

events

dataset powers the Issue page and other features; search and aggregation functions are provided by Snuba. discover dataset supports all Performance Monitoring features. sessions dataset backs the Releases page, ingesting large volumes of pre‑aggregated data for fast queries. outcomes dataset feeds the Stats page.

Getting Started with Snuba

To quickly launch Snuba in a Sentry development environment, run: sentry devservices up --exclude=snuba Ensure the ClickHouse endpoint ( CLICKHOUSE_HOST, default localhost) and the Redis instance ( REDIS_HOST, default localhost on port 6379) are reachable. All services assume UTC time.

Configure Sentry to use Snuba by adding the following lines to ~/.sentry/sentry.conf.py:

SENTRY_SEARCH = 'sentry.search.snuba.EventsDatasetSnubaSearchBackend'
SENTRY_TSDB = 'sentry.tsdb.redissnuba.RedisSnubaTSDB'
SENTRY_EVENTSTREAM = 'sentry.eventstream.snuba.SnubaEventStream'

Then start all services:

sentry devservices up

Snuba Architecture Overview

Snuba

is a time‑series‑oriented data storage service backed by ClickHouse, a column‑oriented distributed database well‑suited for the query patterns required by Snuba.

Data is stored in ClickHouse tables and materialized views, using different ClickHouse table engines depending on the table’s purpose.

Storage

ClickHouse was chosen for its real‑time performance, distributed and replicated nature, flexible storage engines, and consistency guarantees, providing a balanced foundation for Snuba’s needs.

Ingestion

Snuba does not expose an API for row insertion; data is loaded from Kafka topics by a series of consumers that write to ClickHouse tables. Ingestion is most efficient in batches, and the consumer guarantees that each batch of events from Kafka is delivered to ClickHouse at least once. By selecting appropriate ClickHouse table engines, duplicate rows can be deduplicated, enabling exactly‑once semantics under eventual consistency.

Query

Simple point‑in‑time queries are expressed in the SnQL language and sent via an HTTP POST. The query engine translates SnQL into ClickHouse SQL and executes it. Streaming queries use a subscription engine: clients register a streaming query via an HTTP endpoint, the subscription consumer reads from ClickHouse tables, writes results to a Kafka topic, and the client receives updates.

Data Consistency

Snuba defaults to eventual consistency because ClickHouse operates as a multi‑leader system; queries may hit any replica, which may not be the latest. Strong consistency for a specific query can be forced by using ClickHouse’s FINAL keyword, ensuring the query reads from a single, consistent replica and providing sequential consistency.

Snuba in a Sentry Deployment

This section explains Snuba’s role within a full Sentry deployment. Deploying Snuba alone is not useful without the surrounding Sentry services.

Errors and Transactions Data Flow

The single events Kafka topic is shared between the Errors and Transactions pipelines. Errors consumers read from the events topic and write to the ClickHouse errors table, also recording offsets in the snuba-commit-log topic. A synchronized consumer reads both the events and snuba-commit-log topics to generate alerts, writing results to a result topic. Transactions follow an analogous, independent pipeline.

The Errors pipeline includes an additional step: after writing to the errors table, mutation messages are sent to a replacements topic, which the Replacement Consumer processes.

The events topic is partitioned by Sentry project ID to allow ordered processing of alerts and replacements.

Sessions and Outcomes

Sessions enhance the Release Health feature, while Outcomes feed data to the Sentry statistics page. Both pipelines have their own Kafka topics, consumers, and ClickHouse tables.

Change‑Data‑Capture Pipeline

This pipeline is under construction. It consumes a cdc topic and populates two independent ClickHouse tables.