Turn a Generic LLM into an E‑Commerce Risk Detector with Prompt Engineering

1. Introduction: When an Algorithm Engineer Meets an Unpredictable AI

The engineer faced hundreds of risk clusters derived from user behavior embeddings and needed an efficient, consistent way to assess them, leading him to experiment with a large language model (LLM) as a potential assistant.

2. Phase One: From 0 to 1 – Giving the AI an Operations Manual

Key actions:

Role‑Playing : "You are a senior e‑commerce risk‑control expert…" to set the AI’s identity.

Defining Dimensions : Explicitly list analysis dimensions such as recipient info, address analysis, and product‑value analysis.

Structured I/O : Use CSV for input efficiency and require strict JSON output for easy downstream parsing.

The resulting V1 prompt produced structured reports but still suffered a high false‑positive rate.

3. Phase Two: Inject Business Knowledge – “Specific Problem, Specific Analysis”

Added exemption rules and background knowledge to avoid common misclassifications.

Challenge 1 – High discount ≠ risk: clarified that many orders are new‑user first‑orders with platform subsidies.

Challenge 2 – Random strings ≠ fake name: explained that system‑generated IDs are harmless; focus on the actual recipient name.

Challenge 3 – Zero‑price items, nicknames, benefit products are normal and should not be flagged.

Result: false‑positive rate dropped dramatically, elevating the AI to a “mid‑level analyst”.

4. Phase Three: Deepen Analysis – Teach the AI to Think Like a Detective

Introduced concepts of “behavior fingerprints” and “hard links”.

Bottleneck 1 – Ignoring low‑value high‑volume items: expanded risk definition to include large‑scale low‑price goods as potential arbitrage signals.

Bottleneck 2 – Lack of consistency view: added a “shopping‑cart consistency” rule to link accounts with identical or highly similar purchase lists.

Result: the AI could now associate multiple accounts and detect organized fraud, acting as a “senior analyst”.

5. Phase Four: Final Evolution – A Judge‑Style Decision Framework

Implemented a dual‑hypothesis framework (collaborative risk gang vs. benign cohort) and required the AI to search for “hard links” as decisive evidence.

“Hard links are decisive evidence that different accounts point to the same entity, e.g., identical non‑public shipping addresses. If found, classify as risk gang; otherwise, assess if behavior can be explained by legitimate marketing.”

Provided few‑shot examples for both scenarios, turning the prompt into an expert system capable of nuanced, evidence‑based judgments.

6. Summary and Takeaways

Key principles: start by mimicking expert thinking, abstract rules into reusable frameworks, enrich rules with business context, use counter‑examples as teaching tools, and evolve prompts from simple instructions to a full thinking model.