Avoid the 3 Common AI Product Management Pitfalls: Prompt Engineering, RAG, and Fine‑Tuning

Recent conversations with peers revealed a shared headache: which AI technique should be chosen—quick prompt engineering, a RAG knowledge‑base system, or a dedicated fine‑tuned model? The author illustrates the dilemma with two concrete failures.

Case Study 1: E‑commerce Customer Service

An e‑commerce firm spent ¥300,000 to fine‑tune a custom customer‑service model. The model performed well initially, but every promotional campaign changed product rules, causing the model to “forget” and produce wrong answers. Retraining was too slow and expensive, so the investment yielded no business benefit.

Case Study 2: Law Firm Knowledge Assistant

A law firm tried to replace manual case analysis with a prompt‑engineered generic model. Although the model could generate fluent text, it spewed “correct‑but‑useless” statements and hallucinated on complex legal concepts, leading to dangerously inaccurate advice.

Myth‑Busting

Myth 1 – Fine‑tuning = best performance: Fine‑tuning solidifies behavior and works only for stable, rule‑based scenarios (e.g., fixed approval criteria). In dynamic settings like e‑commerce, frequent rule changes make fine‑tuning a maintenance burden.

Myth 2 – RAG is low‑tech: Building a reliable RAG system requires robust document ingestion, chunking, and high‑precision retrieval. Poorly indexed PDFs or scanned images turn RAG into “garbage‑in‑garbage‑out.”

Myth 3 – Prompt engineering is a temporary hack: Prompt engineering is the most flexible, low‑cost way to prototype a new requirement. A few days of prompt work can validate 80 % of use cases without heavy investment.

Five‑Layer Decision Framework ("Five‑Step Thinking")

Business Scenario & Pain Point: Define the exact problem. Is the use case stable or rapidly changing? What accuracy is required?

Data Foundations: Assess data volume, quality, and structure. Small labeled sets favor prompting; large document libraries favor RAG; abundant high‑quality labeled data (> 1,000 examples) enable fine‑tuning.

Cost‑Benefit Analysis: Estimate total cost of ownership (TCO). Prompt engineering: a few thousand yuan; RAG: ¥50‑150 k for knowledge‑base setup and maintenance; Fine‑tuning: ¥100‑300 k plus recurring retraining costs. Quantify benefits (e.g., 30 % labor reduction in a 20‑person support team saves ¥720 k annually).

Technical Resources & Capability: Verify team expertise. Prompting can be done by product managers; RAG needs backend and data‑engineering support; fine‑tuning demands algorithm engineers and annotation resources.

Risk & Red‑Line Assessment: Identify deal‑breakers such as lack of usable data, misaligned stakeholder expectations, or unrealistic timelines. Any red‑line automatically disqualifies the option.

The framework is applied to three representative scenarios.

Scenario A – Enterprise Smart Customer Service

Dynamic product info and frequent rule changes make prompt engineering insufficient and fine‑tuning costly. RAG, with a continuously updated knowledge base, offers the best ROI (≈ 15 % of labor cost for a ¥100 k knowledge‑base investment).

Scenario B – Medical Report Generation

Stable reporting standards and a need for > 95 % accuracy favor fine‑tuning. The hospital already has tens of thousands of high‑quality reports, making the high upfront cost worthwhile for long‑term efficiency gains.

Scenario C – Quick Text‑Classification Proof‑of‑Concept

Limited data and a tight budget point to prompt engineering. A prototype can be built in days for a few thousand yuan, allowing rapid validation before any larger spend.

Advanced Outlook – Agentic RAG

Traditional RAG follows a single “question → retrieve → answer” flow, which breaks on multi‑step problems. Agentic RAG introduces an autonomous “agent” that decides when and what to retrieve, performs multi‑round retrieval, calls external tools (e.g., flight‑price APIs), and may ask follow‑up questions. This enables complex tasks such as planning a three‑day Shanghai‑Beijing trip within a ¥5,000 budget.

Action Checklist for AI Product Managers

Start from the business pain point, not the technology buzzword.

Validate data quality yourself; data quality caps AI performance.

Prototype with prompt engineering first; only scale up after measurable success.

Treat RAG as an evolving knowledge product, not a one‑off project.

Define clear evaluation metrics and track them continuously.

The overarching lesson is that the best solution is the one that solves the business problem at the lowest total cost, not the flashiest technology.