Why AI Product Managers Struggle with Planning: Insights from Real Interviews

I have interviewed several AI product managers and noticed a striking pattern: most can discuss AIGC, multimodal models, and agents fluently, yet when asked, "How would you design an evaluation system to verify your product's value?" they freeze.

They often answer with generic suggestions like "collect user feedback" or "run A/B tests," which only scratch the surface. For AI products, evaluation is not an after‑thought but a mandatory component that spans the entire product lifecycle—from definition through development to iteration.

Why "no evaluation, no AI"? The author shares a personal failure: two years ago they led an intelligent‑customer‑service chatbot project that used a state‑of‑the‑art BERT model and achieved 95% offline accuracy. After launch, user satisfaction plummeted, and complaints flooded in, exposing a classic "high‑score‑low‑impact" problem. The issue was that the sole metric—accuracy on a clean test set—ignored multi‑turn dialogue handling, intent understanding, and guidance under incomplete information.

From this experience the author distilled three core values of a good evaluation system:

Direction: It acts as a compass, telling the team whether to improve model creativity, instruction compliance, hallucination reduction, or knowledge freshness.

Quantify Progress: It enables precise statements such as "Model v2 improves factual accuracy by 15% but reduces fun by 5%," guiding product decisions.

Build Trust: It provides a solid shield for stakeholders—customers, managers, and the market—to believe in the AI's reliability.

The author proposes a "1+3" multi‑dimensional evaluation framework:

1 (Core): User value is the ultimate north‑star. Every metric must trace back to whether the product creates user value.

3 (Dimensions):

Offline Evaluation: Large‑scale lab tests on fixed datasets (the "mock exam"). Fast, cheap, but may miss real‑world nuances.

Online Evaluation: A/B tests on live traffic (the "final exam"). Highly persuasive but slower and riskier.

Human‑in‑the‑Loop & Adversarial Testing: Expert judges assess creativity, empathy, and safety; red‑team attacks expose hidden biases.

The workflow for a new model version (e.g., Model v2.1) follows a closed loop:

Run offline evaluation; if basic scores drop below the current baseline, reject early.

Pass promising candidates to human evaluation, where product, ops, and domain experts score soft attributes (creativity, empathy) and red‑team probes safety.

Only models that excel in both offline and human stages proceed to online A/B testing with a small traffic slice (e.g., 5%).

The winner becomes the new baseline model.

To operationalize this, the author introduces a three‑layer "funnel" metric system:

First Layer – North Star Metric

Aligns with business goals (e.g., subscription renewal rate for a writing assistant, GMV for a recommendation engine, or problem‑resolution rate for a chatbot).

Second Layer – User‑Experience / Product Metrics

Adoption rate (percentage of generated content that users copy, export, or publish).

Task success rate (how often users achieve their goal with the AI).

User satisfaction score (1‑5 star feedback after each interaction).

Interaction rounds / duration (fewer turns indicate higher efficiency).

Third Layer – Model‑Performance / Technical Metrics

Relevance & instruction compliance.

Accuracy & factuality (detecting hallucinations).

Fluency & coherence.

Creativity & diversity.

Safety & value alignment.

Domain‑specific metrics (e.g., code execution rate, image generation aesthetics, speech naturalness).

These layers form a hypothesis chain: improving factuality (third layer) should raise adoption (second layer) and ultimately boost the north‑star metric (first layer).

Building the Evaluation Set – the "ruler" for the metric system. Sources include:

Live user logs (gold‑mine of real queries).

Manually crafted cases by product or domain experts.

Public benchmarks (SuperGLUE, MMLU, etc.).

AI‑generated data (e.g., using GPT‑4 to simulate tricky user requests).

A robust evaluation set must be comprehensive, representative, and capable of surfacing model bias.

Designing the Evaluation Set Matrix – not a single dataset but multiple matrices:

General‑ability sets covering common intents.

Domain‑specific sets for verticals like medical or legal.

Probe sets targeting single abilities (e.g., math reasoning).

Adversarial / safety sets to test robustness.

For subjective metrics (creativity, humor), the author stresses clear annotation guidelines and consensus methods (multiple annotators, expert arbitration, Fleiss’ Kappa for consistency).

Practical Walk‑through – the author demonstrates the framework with a short‑video script AI Agent called "Script Genie":

North‑star: Script adoption rate (copy/export actions).

User‑experience metrics: First‑effective‑script generation time, modification rate, satisfaction score.

Technical metrics: Instruction compliance, content creativity, structural completeness, audiovisual language richness, "viral" potential score, and safety.

The same 1+3 workflow—offline scoring, human review, online A/B—applies, ensuring each model iteration is validated before full rollout. Overall, the article provides a systematic, data‑driven methodology for AI product managers to design, implement, and iterate evaluation systems that align technical performance with real user value.