How We Built an LLM‑Powered User Feedback Sentiment Monitoring System

Problem Statement

Manual inspection of massive, multi‑channel user feedback in the transaction domain suffers from four critical issues:

Low efficiency : Scanning millions of feedback items consumes excessive time and labor.

Missed reports : High‑risk or early‑stage problems can be buried in the data flood.

Trend blindness : Detecting emerging issues or sudden spikes is difficult without systematic analysis.

Missing closed‑loop : After a problem is handled, there is no systematic tracking or post‑mortem.

End‑to‑End AI Workflow

The solution is a fully automated pipeline that transforms raw feedback into actionable alerts and attribution records:

Collect → Clean → AI Judgment → Alert → Distribute → Attribution → Review

Each stage is described below.

Data Collection & Cleaning

Feedback is harvested from the internal "吐槽吧" channel. An initial filter runs on the MagicRabbit platform to remove obvious non‑feedback items. A secondary keyword‑based extraction (mirroring the manual inspection rules) re‑captures any missed entries. The resulting records are normalized to a minimal schema (e.g., {"content": "...", "module": "...", "timestamp": "..."}) before feeding the AI model.

Core AI Model Pipeline

The model follows a four‑step "recognize → judge → match → learn" process:

Identify key elements : The LLM extracts the "who/what" and the event description, e.g., "rating page" + "cannot open".

Determine intent & sentiment : The extracted text is classified into suggestion , bug , or question , and a polarity label (positive/negative) is assigned.

Semantic match against a curated taxonomy : The intent‑sentiment pair is compared with a manually built problem‑category knowledge base. The knowledge base contains separate taxonomies for each business line (evaluation, logistics, order, payment, etc.) with representative keywords and example sentences. Matching is constrained to existing categories; the model cannot invent new labels.

Closed‑loop learning : Feedback that fails to match any category is routed to a periodic enrichment process (see below).

Knowledge Base & Taxonomy

The taxonomy is constructed offline by domain experts who label historical feedback. Each category includes:

A unique identifier.

Typical trigger keywords.

Example feedback snippets.

During inference the LLM performs a similarity search (e.g., embedding‑based cosine similarity) between the input feedback and the category prototypes, returning the best‑matching category ID.

Closed‑Loop Learning Mechanism

Unmatched feedback is not discarded. Instead, a scheduled job runs (e.g., nightly) that:

Clusters the unmatched items using a large‑model embedding space.

Runs topic discovery to surface recurring themes.

Presents the clusters to business analysts for validation.

When a new theme is approved, it is added to the taxonomy as a new category, together with its keyword set.

This feedback loop continuously enriches the knowledge base, reduces the unmatched rate, and improves overall classification precision.

Alert & Distribution Mechanism

New‑issue alert : Triggered when a problem type appears for the first time on a given day or when its absolute count exceeds a configurable threshold.

Spike alert : Compares the current daily count with a historical baseline (default: 7‑day moving average). An alert fires if the growth ratio exceeds a user‑defined percentage (e.g., 50 % or 100 %).

DingTalk sync : At a configurable time each day, alerts are pushed to the relevant DingTalk groups. The message includes problem type, current count, growth rate, a representative feedback excerpt, and a quick‑jump URL to the detailed view.

Visualization Dashboard

A multi‑dimensional dashboard visualizes:

Total feedback volume and day‑over‑day trends.

Distribution of problem types across business modules and platform versions.

Active alert list with drill‑down links.

Raw feedback details for ad‑hoc analysis.

Users can filter by time range, module, category, or version to investigate specific patterns.

Iterative Development History

Batch feeding (free classification) : Early prototypes fed large batches of raw feedback to the LLM for unconstrained classification. This worked on small datasets but hit token limits and produced mismatched output counts.

Per‑item feeding (stability) : Feedback was structured and sent one‑by‑one, improving accuracy but exposing the probabilistic nature of LLM outputs—identical semantics sometimes yielded different categories.

Pre‑label + semantic matching (controlled categories) : A manually curated taxonomy was introduced, and the LLM’s role shifted to semantic matching against this fixed set. This guarantees that all outputs stay within predefined categories, delivering high consistency and interpretability.

The final architecture balances classification accuracy, result consistency, and maintainability, while providing an explainable taxonomy that can be iteratively refined.

Future Outlook

The system is intended as an assistive decision‑making tool, not a black‑box replacement for human judgment. Ongoing work focuses on:

Improving classification accuracy through prompt engineering and model fine‑tuning.

Extending the taxonomy to cover additional business scenarios.

Automating knowledge‑base maintenance (e.g., auto‑suggested keywords from high‑confidence clusters).

Strengthening user trust by exposing confidence scores and audit trails.

Key Visuals

The following images illustrate the workflow, taxonomy matching, alert rules, and dashboard layout.