Content Embedding Practices and Challenges at Hulu

Hulu, a leading US video‑streaming platform, structures its service around users, content, and ads, and seeks to represent each piece of content as a dense vector (content embedding) to support downstream algorithms.

The platform defines four hierarchical levels of content understanding: video‑level detection, episode‑level tagging, collection‑level relationships, and lifecycle interaction analysis.

Three primary representation types are discussed: ID‑based (one‑hot), tag‑based (keyword lists), and vector‑based embeddings, with the latter being the focus of the content‑embedding effort.

Initial tag‑based embeddings are generated by constructing a content‑attribute graph linking shows and their attributes, then applying node2vec to learn embeddings for both show and attribute nodes.

Limitations of this approach include reliance solely on tag data, incomplete coverage of key tags, lack of semantic weighting for tags, and insufficient interaction between different tag types.

To incorporate non‑tag textual metadata, two methods are explored: directly embedding text using BERT + SIF, and a second method called Metadata‑BERT that predicts additional tags from text, enriching the graph before re‑embedding, achieving 83% tag‑prediction accuracy.

Multimodal embeddings are also considered: visual frames and audio are processed with models such as Inception/VGGish, then aggregated via NeXtVLAD or BERT‑style encoders to produce a unified content vector.

Applications of the resulting embeddings include computing content similarity for recommendation (e.g., "You May Also Like"), serving as features in TV/movie ranking models, improving cold‑start retrieval by adding a content‑based recall branch, and aiding automated collection creation and expansion.

The article concludes with open questions about evaluating embedding quality, bridging the gap between offline embedding training and online business objectives, and the potential for end‑to‑end task‑specific embedding learning.