Why a Simple Workflow Beats Complex Agents in AI‑Powered Insurance Audits

Introduction

This article reviews a year‑old AI audit project for an insurance claim scenario, highlighting how a carefully designed workflow and prompt engineering outperformed more complex agent‑based approaches.

Business Background

Taobao’s "Juhui Home" service offers large‑item freight insurance, breakage coverage, and a three‑year warranty. Human auditors review order information, logistics data, and uploaded evidence to decide whether a claim is valid.

AI Audit Design

The AI system follows a three‑step process: (1) identify the product category, (2) describe the image content and possible damage, and (3) output a structured JSON verdict with isBroken, description, and position. Prompt templates were refined to include explicit reasoning steps and output constraints.

Key Findings

Simple rule‑based filters (e.g., amount limits, policy validity, risk rules) can reject about 20% of invalid claims before invoking the model.

When the claim amount is within limits, 80% of the remaining cases are approved, suggesting that model accuracy is less critical than proper pre‑filtering.

Prompt engineering that breaks the task into sub‑steps dramatically improves consistency. Adding a description of possible damage types and asking the model to list its reasoning reduces hallucinations.

Multi‑image inputs (product photos, packaging, logistics receipts) provide valuable context; treating them jointly yields better judgments than single‑image analysis.

Experimental Results

Using the refined prompts, the LLM achieved 89% accuracy, 96.9% precision, 91.3% recall, and an F1 score of 93.99% on the test set. A baseline model that always approves scored 85% across all metrics, confirming the benefit of the AI workflow.

Further experiments showed that encouraging users to annotate damage locations on images improves detection of subtle defects such as scratches or dents.

Technical Improvements

Compress uploaded images to stay under the provider’s size limits (≈10 MB) and avoid request throttling.

Separate the detection and verification stages: use a vision model for object detection, then a language model for logical consistency checks, optionally with different temperature settings.

Iteratively adjust prompts based on error analysis (e.g., treating scratches as damage) to raise recall without sacrificing precision.

Future Directions

As multimodal models become more capable, the workflow can evolve into a three‑stage pipeline: (1) target detection, (2) detailed content analysis, and (3) cross‑validation with an independent model. This reduces reliance on fragile agent loops and improves interpretability.

Team Introduction

The author, Xin‑Ning, works in the Financial Technology Department of TaoTian Group, focusing on building large‑scale financial ecosystems and applying AI to real‑world scenarios for millions of merchants and consumers.