How Goldsky Builds a Gold‑Standard Web3 Real‑Time Data Pipeline with ClickHouse and Redpanda

Introduction

Goldsky, an open‑source‑rooted company, shares the architecture it uses to deliver cryptocurrency data from more than 15 popular blockchains to customers. By exposing a public ClickHouse instance with data from the Base blockchain, Goldsky enables developers to create proof‑of‑concepts without managing ingestion pipelines or infrastructure costs.

Goldsky

Goldsky provides crypto data services, delivering indexed blockchain data, subgraphs, and data streams to its own storage. It offers a "data‑pipeline‑as‑a‑service" that lets users filter and transform blockchain data before exposing simple APIs, avoiding the complexity of RPC providers. For analytical workloads, ClickHouse is the preferred database, requiring efficient delivery of TiB‑scale subsets to potentially thousands of ClickHouse clusters.

Architecture

The stack consists of Redpanda, Apache Flink, and ClickHouse. Data is pushed from indexers into Redpanda topics (one per data type). Flink consumes these topics, applies user‑written FlinkSQL transformations, and writes the results into ClickHouse for analysis. The multi‑tenant design allows Goldsky to serve any crypto dataset to thousands of ClickHouse clusters via a simple API.

Note: The subgraph module, a single‑threaded indexer that runs TypeScript‑written WebAssembly logic, can perform custom aggregations and additional HTTP calls to Ethereum during indexing; its output can be exposed via API or inserted into ClickHouse.

Using Redpanda as Backend Storage

Goldsky stores transformed blockchain data in schema‑driven Avro format on Redpanda. The main challenges are infinite retention, keeping data up‑to‑date, and delivering it with minimal end‑to‑end latency. Redpanda was chosen over Kafka because its tiered‑storage architecture stores data cost‑effectively on object storage while delivering GiB/s throughput, and recent Jepsen tests confirm its high durability, matching Goldsky's pattern of focusing on recent blocks.

Processing with Apache Flink

FlinkSQL provides users with filtering, stream‑side joins, Top‑N counting, and pattern recognition while maintaining high‑performance inserts into ClickHouse (up to 500k events/second). The article references a blog post that explains handling blockchain re‑orgs efficiently.

Analyzing with ClickHouse

When large‑scale query analysis is required, ClickHouse becomes the primary choice due to its unmatched query performance, cost efficiency, and enhanced SQL support for TiB‑scale crypto datasets. Services like Dune have popularized SQL queries over blockchain data.

Back‑filling with ClickHouse

Goldsky uses a hybrid source that consumes from ClickHouse for back‑fills and from Redpanda for incremental updates. Aggregations defined in the pipeline can run across both systems transparently to the user.

Challenges and Lessons Learned

The biggest challenge was mastering the ReplacingMergeTree engine to handle updates efficiently. Goldsky leveraged pre‑where simulation, partitioning for fast queries, and recent control over the FINAL operator thread count. Users can also customize the ORDER BY key (e.g., block timestamp or address) to match access patterns.

Example Dataset

Goldsky selected the Base blockchain (an Ethereum L2 with a hybrid Proof‑of‑Stake/Proof‑of‑Work consensus) as a high‑volume test case. The public ClickHouse instance provides tables for blocks, logs, transactions, and traces, ranging from tens of millions to over a billion rows (up to ~1 TiB uncompressed).

SELECT
    table,
    formatReadableQuantity(sum(rows)) AS total_rows,
    round(sum(rows) / 42) AS events_per_day,
    formatReadableSize(sum(data_compressed_bytes)) AS compressed_size,
    formatReadableSize(sum(data_uncompressed_bytes)) AS uncompressed_size,
    round(sum(data_uncompressed_bytes) / sum(data_compressed_bytes), 2) AS ratio
FROM system.parts
WHERE (database = 'base') AND active
GROUP BY table
ORDER BY sum(rows) ASC;

Typical analytical queries run in sub‑second latency, e.g., daily transaction counts with a 7‑day moving average.

SELECT
    toStartOfDay(block_timestamp) AS day,
    COUNT(*) AS txns,
    ROUND(AVG(txns) OVER (ORDER BY day ASC ROWS BETWEEN 6 PRECEDING AND CURRENT ROW)) AS `7d avg`
FROM base_transactions
WHERE day > '2023-07-12'
GROUP BY 1
ORDER BY 1 DESC;

Conclusion

The article explains why Goldsky chose Redpanda, Apache Flink, and ClickHouse for a multi‑tenant, real‑time blockchain analytics service. Stream processing naturally fits blockchain data, and ClickHouse adds fast, subset‑or‑full‑chain query capabilities. The combination delivers a robust, community‑focused analytics platform, with a free public dataset for Base and plans to open more blockchains.