Inside Snuba: How Sentry’s Query Processing Pipeline Transforms Queries

Snuba Query Processing Pipeline

Snuba parses a query written in the legacy JSON‑based language or the newer SnQL language into an AST, then executes a ClickHouse SQL query. Between parsing and execution a series of passes transform the AST to optimise the query and to reject queries that could endanger the infrastructure.

Logical Processing

Query Validation

Validation returns HTTP 400 with an error message for invalid queries. It consists of two stages:

General validation performed in QueryEntity (e.g., alias shadowing checks, function‑signature validation). Source: https://github.com/getsentry/snuba/blob/master/snuba/query/parser/__init__.py#L91

Entity‑specific validation defined in each entity class (e.g., requiring a project_id condition). Source: https://github.com/getsentry/snuba/blob/master/snuba/datasets/entity.py#L46-L47

Logical Query Processors

Stateless AST transformers modify the query in place. Each entity registers an ordered list of processors. Examples:

Custom function apdex – https://github.com/getsentry/snuba/blob/10b747da57d7d833374984d5eb31151393577911/snuba/query/processors/performance_expressions.py#L12-L20

Time‑series processor – https://github.com/getsentry/snuba/blob/master/snuba/query/processors/timeseries_processor.py

Processors must be independent of one another.

Storage Selector

Each entity can define multiple storages (tables or materialised views). The selector chooses the appropriate storage for a query. Example: the Errors entity has a primary storage routed to write nodes and a read‑only replica to reduce load. Source: https://github.com/getsentry/snuba/blob/master/snuba/datasets/storage.py#L155-L165

Query Translator

After storage selection the query is translated into a physical ClickHouse query. Translation is rule‑based; each storage defines a list of translation rules applied sequentially. Rules operate on single expressions because ClickHouse’s join engine does not perform expression push‑down. Example translation rules for the transactions entity: https://github.com/getsentry/snuba/blob/master/snuba/datasets/entities/transactions.py#L33-L81

Physical Query Processors

Physical processors operate on the ClickHouse query to improve performance. Example: the mapping optimizer replaces tag equality checks with tag‑hash lookups using Bloom‑filter indexes, speeding up filtering. Source: https://github.com/getsentry/snuba/blob/master/snuba/query/processors/mapping_optimizer.py

Query Splitter

Some queries are split into multiple ClickHouse queries and their results are combined. Two split strategies are implemented:

Time splitting – a non‑aggregated, sorted query is divided into progressively larger time windows until enough results are obtained.

Column splitting – the filter is executed on a minimal column set first; a second query fetches the remaining columns for the rows selected by the first query.

Implementation: https://github.com/getsentry/snuba/blob/master/snuba/web/split.py

Query Formatter

The final component formats the processed query into a ClickHouse query string.

Composite Query Processing

For queries that involve joins or subqueries the pipeline adds extra steps.

Subquery Generator

The generator takes a SnQL join query and creates a subquery for each table involved.

Expressions Push‑Down

A join optimizer pushes expressions that can be evaluated in subqueries down into those subqueries because ClickHouse’s join engine does not perform expression push‑down.

Simple Query Processing Pipeline

The generated subqueries then pass through the same logical‑to‑physical pipeline described above.

Join Optimizations

After processing, additional optimisations such as converting a regular join into a Semi Join are applied.