Chronon: Open‑Source Feature Platform for Machine Learning – Architecture, Workflow, and Code Examples

Chronon is an open‑source feature platform that provides observability and management tools, allowing machine‑learning practitioners to ingest various data sources while abstracting the complexity of data engineering. It delivers high‑throughput, low‑latency stream processing and supports both batch and real‑time feature computation.

Background

Practitioners often spend most of their time managing data pipelines rather than building models. Traditional approaches either copy offline data to online environments (risking inconsistency and label leakage) or record online data and wait for enough history before training (causing long latency). Chronon lets users define a feature once and automatically generates both offline (training) and online (inference) pipelines, with strong observability, data‑quality checks, and feature sharing.

How it works – Feature definition

The platform uses a declarative API to describe sources, aggregations, and joins. Below is a simplified example that defines purchase‑related features from an event source.

source = Source(
    events=EventSource(
        table="data.purchases",  # Log table in the warehouse (batch updated daily)
        topic="events/purchases",  # Streaming source topic
        query=Query(
            selects=select("user_id", "purchase_price"),
            time_column="ts"
        )
    )
)

window_sizes = [Window(length=day, timeUnit=TimeUnit.DAYS) for day in [3, 14, 30]]  # Define windows

v1 = GroupBy(
    sources=[source],
    keys=["user_id"],
    online=True,
    aggregations=[
        Aggregation(input_column="purchase_price", operation=Operation.SUM, windows=window_sizes),
        Aggregation(input_column="purchase_price", operation=Operation.COUNT, windows=window_sizes),
        Aggregation(input_column="purchase_price", operation=Operation.AVERAGE, windows=window_sizes),
        Aggregation(input_column="purchase_price", operation=Operation.LAST_K(10))
    ]
)

A similar definition creates user‑profile features directly from a snapshot table without aggregation.

source = Source(
    entities=EntitySource(
        snapshotTable="data.users",
        query=Query(
            selects=select("user_id", "account_created_ds", "email_verified")
        )
    )
)

v1 = GroupBy(
    sources=[source],
    keys=["user_id"],
    aggregations=None,
    online=True
)

Features are then combined with a Join operation to produce a single view usable for both back‑fill (offline) and real‑time inference.

source = Source(
    events=EventSource(
        table="data.checkouts",
        query=Query(
            selects=select("user_id"),
            time_column="ts"
        )
    )
)

v1 = Join(
    left=source,
    right_parts=[JoinPart(group_by=gb) for gb in [purchases_v1, returns_v1, users]]
)

Back‑fill / Offline computation

Chronon can replay historical events with precise timestamp alignment, handling data skew and optimizing compute efficiency. Batch jobs generate feature values for past windows, which are then materialized in an online KV store.

Online computation

For batch‑based features (e.g., user profile), a daily batch job updates the KV store. For streaming features (e.g., purchase aggregates), Chronon runs both a batch upload and a real‑time stream job that continuously updates the KV store with the latest values.

Feature retrieval API

Chronon exposes a low‑latency API to fetch features for a given key. Example in Java:

// Fetching all features for user=123
Map
keyMap = new HashMap<>();
keyMap.put("user", "123");
Fetcher.fetch_join(new Request("quickstart_training_set_v1", keyMap));
// Sample response: {"purchase_price_avg_3d":14.2341, "purchase_price_avg_14d":11.89352, ...}

And the equivalent Python CLI command:

run.py --mode=fetch -k '{"user_id":123}' -n quickstart/training_set -t join
> {"purchase_price_avg_3d":14.2341, "purchase_price_avg_14d":11.89352, ...}

These APIs can be wrapped in REST services for use in non‑Java environments.

Online‑offline consistency

Chronon measures consistency by logging each feature‑fetch request (key and timestamp), replaying the request through the join back‑fill, and comparing the back‑filled value with the live value.

Future directions

The open‑source release is the first step toward a collaborative ecosystem with Stripe and the broader community. Planned improvements include reducing iteration and compute costs, simplifying new feature creation (e.g., via natural‑language‑to‑code generation), and building intelligent agents that suggest model updates, detect data drift, and automate experimentation.