Building a Model-Driven Machine Learning System at Tubi: From Simple A/B Tests to Embedding-Based Recommendations

Tubi TV, one of the largest free video platforms, relies heavily on machine learning across recommendation, search, advertising, and user acquisition, making ML the core of its business.

Measure First – Before chasing complex deep or reinforcement learning models, the team advises building a simple, end‑to‑end experiment system (A/B testing) to validate ideas.

Why? Because algorithms that work for some users may fail for others, and thorough A/B testing provides unbiased feedback before deployment.

Define a Launch Plan – Successful A/B testing requires a clear action plan that specifies one or two core metrics for go/no‑go decisions and secondary metrics for iteration. If core metrics are significantly positive (or neutral with positive secondary metrics), the feature is launched; otherwise it is not.

Simple Models: Popularity + Embedding – Early on, a basic popularity (heat) ranking based on recent watch time (e.g., last day vs. last week) works well, provided artificial boosts from promotions are removed. Additionally, generating embeddings for videos (using Spark's Word2Vec or ALS) enables similarity‑based ranking for users, platforms, locations, etc.

Once video embeddings are available, other entities (users, genres, postal codes) can also be embedded, allowing cosine‑similarity calculations for diverse ranking problems.

Iterate Fast – After establishing basic algorithms, the team focuses on rapid iteration rather than immediately pursuing cutting‑edge techniques like reinforcement learning. Frequent, low‑cost experiments (up to 50 launches per year in a small team) drive continuous improvement.

In summary, Tubi’s experience shows that a simple, well‑measured, and iteration‑focused ML pipeline—starting with basic A/B testing, clear launch criteria, and lightweight models—can scale to support a large streaming service.

The article concludes with an invitation to join Tubi’s growing team, listing open senior engineering positions across data, backend, mobile, and frontend development.