Monitoring Quality Issues in Tubi’s Recommendation System

Tubi is an ad‑supported video‑on‑demand (AVOD) streaming service that offers movies, TV shows, live news, and sports across web, mobile, and OTT platforms. Its recommendation system is a core component that helps users discover the next video they want to watch, making reliable monitoring essential.

Where Quality Problems Appear

The system can degrade even while running, unlike traditional services that are simply up or down. Potential issues include abnormal user information (Garbage‑in‑Garbage‑out), errors in post‑processing steps (e.g., expired video detection), and problems in the model update pipeline such as missing content metadata, context feature errors, label imbalances, and upstream data quality.

Monitoring Solution

To address these challenges, Tubi monitors three key data sources:

Data passed to the model for prediction.

The final recommendation results displayed to the client.

The training data that determines model quality.

Model‑input monitoring records client‑side data (e.g., language settings) to catch anomalies early. Visualizations of language‑percentage trends helped identify and resolve platform‑specific bugs.

Final recommendation monitoring tracks attributes such as the percentage of each video type in the recommendation list; unusually low content counts signal system‑wide issues.

Training‑data monitoring watches row counts, positive‑sample ratios, and feature‑missing‑value rates to ensure the training set remains consistent with expectations.

Design and Considerations

The monitoring platform emphasizes timeliness (real‑time detection of request/response streams), data characteristics (different handling for real‑time event streams vs. batch updates), and scalability (independent metrics, horizontal scaling, and support for new model types).

Metrics are defined by a flexible protocol consisting of data source, scope, and metric name. Users can create metrics via Scala, Python APIs, or a DSL that uses JSON‑path expressions. Configuration files, rather than job redeployments, are used to update extraction rules dynamically.

Key Takeaways

Continuous monitoring of recommendation quality is essential; existing metrics may need adjustment and new metrics added as problems evolve.

Ownership of alerts by domain owners ensures precise, actionable notifications.

Prioritizing precision over recall prevents alert fatigue.

Monthly monitoring summaries help teams stay informed about system health.

The article concludes with a recruitment call for engineers interested in high‑impact, large‑scale projects at Tubi.