Recommendation System Optimization: Lessons, AB Testing Cycles, and Practical Principles

The article shares the author’s years‑long experience in recommendation system optimization, describing the evolution from simple cosine‑similarity based collaborative filtering to a wide range of shallow and deep, discriminative and generative machine‑learning models.

It emphasizes that effective optimization depends first on a well‑defined problem and then on an appropriate optimization method, rather than solely on the loss function or algorithmic tricks.

A typical optimization cycle is illustrated as an “onion‑layer” process driven by A/B experiments, highlighting the difficulty of obtaining stable, quantifiable gains and the risk of short‑term success disappearing at scale.

The article points out two scenarios where A/B testing is ineffective: non‑instant‑feedback strategies and system‑flow optimizations that involve multiple interdependent stages.

Four practical principles are proposed: (1) never do fundamentally wrong things; (2) ensure the right things are done correctly; (3) prefer simple, reliable solutions over complex ones; (4) avoid over‑optimization and focus on cost‑benefit trade‑offs.

Ultimately, the author argues that recommendation‑system engineers should concentrate on discovering, abstracting, and defining system problems, then solving them with minimal assumptions and low error‑prone methods, rather than relying on sophisticated algorithms alone.