How Hulu Eliminated Feature Drift with Server‑Side Feature Logging

Effective feature representation is crucial for recommendation models; consistent feature collection prevents training‑serving drift.

In Hulu's recommendation workflow, offline training and online prediction use different data sources, execution times, and code, leading to feature drift.

Feature drift arises from three main sources: (1) data source differences—online features come from real‑time services while offline features are derived from data‑warehouse fact tables (e.g., Jaccard distance 0.28 for TV‑series lists); (2) execution time differences—offline pipelines may use newer feature versions than online caches; (3) code logic differences—offline Spark jobs and online Java services are developed independently, causing mismatches such as unit conversion errors.

To measure and address drift, Hulu built a Server‑Side Feature Logging system composed of four components: Feature Encoder, Data Integration, Feature Consistency Check (FCC), and Dataset Builder.

Feature Encoder encodes online features into binary data and streams them to a platform; Data Integration persists the binary stream into the offline data warehouse; FCC computes drift metrics between online‑collected and offline‑extracted features; Dataset Builder merges online‑collected features with offline labels to create training datasets, ensuring that training and serving use identical features.

The system stores features in a dynamic Map structure, automatically handling new online features without schema changes. Online features are prioritized; if unavailable, offline‑computed features are used, and configuration can force offline usage for problematic features.

Compression is achieved by categorizing feature values, encoding Maps with Avro, and enabling Kafka compression, reducing transmission cost by ~70% (Avro) and an additional 80% (Kafka).

FCC and Dataset Builder employ column pruning, join optimization, windowing, and Bloom filters, cutting resource consumption by about 50% and accelerating dataset generation.

High availability is provided by Ribbon, a Flink‑based integration tool that uses checkpointing and automatic restarts to ensure continuous data flow.

Empirical results show that models trained with online‑collected features consistently outperform those using only offline features, improving user engagement metrics while lowering computational overhead.