Building Fast-Iterating Machine Learning Systems at Tubi: A/B Testing, Simple Models, and Embedding Strategies

Tubi TV, one of the largest free‑video platforms, leverages machine learning across recommendation, search, advertising, and content acquisition, making ML the core of its business.

Measurement First – Before diving into advanced deep or reinforcement learning, the team stresses building a simple, end‑to‑end experiment system (A/B testing) to validate ideas, because complex models can behave inconsistently across user segments.

Experiment Planning – Successful experiments require a concise launch plan: define one or two core metrics for go/no‑go decisions, supplement with secondary metrics for iteration, and agree on a clear rule (e.g., core metrics positive → launch).

Simple Models – Start with lightweight models such as heat‑based ranking (using recent watch time) and similarity‑based ranking via embeddings. Heat scores should exclude artificial boosts from promotions. Embeddings can be generated with Spark's word2vec or ALS, then used for cosine similarity across videos, users, or other entities.

Iteration Is King – After establishing basic algorithms, the team focuses on rapid iteration rather than pursuing the latest research. By running many small experiments (up to 50 per year in early stages), they continuously refine models, prioritize ROI, and keep the system adaptable.

Conclusion – Tubi’s experience shows that a fast, experiment‑driven ML pipeline, clear launch criteria, and simple yet effective models enable rapid product improvements and scalable growth.

Recruitment Invitation – Tubi is hiring senior engineers for data, ML, backend (Elixir, Scala), Android, iOS, frontend, and QA roles; details are on their website.

Event Announcement – An upcoming DataFunTalk big‑data architecture forum will feature a talk by Tubi senior data engineer on data‑quality architecture; registration via QR code.