On-Device Intent Detection at Taote: LoRA, ShortGPT Pruning & AWQ Quantization

Background: Why On-Device Models

Taote began exploring on-device models in March 2024 to address three drivers: (1) continuous pursuit of real-time user intent since 2013, evolving from engineering to neural networks to LLMs; (2) maturity of AI infrastructure — group-wide training environments, base-model fine-tuning, quantization tooling (LLaMA-Factory, AliNNPrivate) and AI coding assistants (Claude Code, Cursor) lowered entry barriers; (3) business needs — identify whether a user has strong shopping intent or is casually browsing, and detect imminent churn to trigger retention coupons, while avoiding cloud LLM latency and token costs.

Target Intents & Scenarios

After iterative experiments, the team defined three intent classes for in-app shoppers:

Browsing (闲逛) : aimless cross-category scrolling.

Price Comparison (比价犹豫) : repeated views within the same leaf category.

Scenario Shopping (场景购物) : cross-sub-category purchases around a unified theme (e.g., camping: tent, canopy, table).

Two deployment scenarios:

Retention coupon : when a user with purchase intent is about to leave, push a coupon to lift conversion.

Feed re-ranking : reorder unexposed feed cards based on detected intent to surface relevant items and improve CTR.

Example behavior sequence (JSON-like log) shows repeated air-conditioner remote control detail-page views; expected model output:

"比价犹豫|空调遥控器;美的;通用款;省电星;华凌"

.

Overall Architecture (4 Layers)

Model Layer : dataset cleaning, LoRA hyper-parameter tuning, quantization parameter search.

Inference Engine Layer : engine internals mastery & end-to-end latency optimization.

On-Device AI Lab : model management & download, user behavior assembly, engine invocation, intent distribution.

Business Layer : intent consumption.

Dataset Construction: Two Generations

Gen-1: From Raw Logs to Synthetic Labels

Cold-start with no on-device inference logs. Pipeline: rule-engine hard filtering → cloud LLM (Qwen3-Max) distillation → human verification → training set. Key decisions:

Discarded CoT : increased hallucination (accuracy <60%) and added ~1 min latency.

Dual-teacher cross-validation : added Kimi K2.5 (72% agreement with Qwen3-Max on 50 stratified samples, better than GLM-5 70%, DeepSeek V3.2 68%). Auto-accept when both agree (69.3% of 10,300 samples in 0420 batch), diverge → human label.

Gen-1 limitations: ceiling equals teacher model rules; label distribution manually set (Scenario 7% / Comparison 15% / Browsing 78%) causing offline/online distribution shift. After multiple iterations: DSW env accuracy 84%, quantized on-device 76%; small-scale canary in early July.

Gen-2: Rebuilding from On-Device Inference Logs

After canary reached 100k users, collected real on-device logs, cleaned and rebuilt dataset matching online intent distribution. Cleaning highlights:

Three-way consistency : on-device output vs. two cloud models. Three-way agreement → high-confidence positive; two clouds agree but on-device differs → hard negative (model's error boundary) → targeted augmentation.

Keyword quality gate : Qwen-Max scores candidate keywords 0–3 for non-browsing samples; ≤1 discarded to prevent hallucinated keywords.

Human annotation : three-model disagreement → blind human labeling (no model predictions shown) to avoid anchoring; limited to 50 samples per round due to cost.

Gen-2 results: DSW accuracy 91%, quantized on-device 80%.

Training Platform & Base Model Selection

Platform criteria: (1) support post-fine-tuning conversion, quantization, download; (2) modern GPU (quantized accuracy varies >21% across GPU generations). Chose ModelScope + LLaMA-Factory after evaluating ideaLab, ModelScope, etc. (some lacked Qwen3-0.6B support, download, or had old GPUs/long queues).

Base model: Qwen3-0.6B (over Qwen2.5-0.5B) due to mature AliNNPrivate/MNN adaptation and internal proven cases.

Prompt Engineering & Ablation

System Prompt includes intent definitions, category hierarchy, decision rules, output format, keyword rules. Content Prompt = user behavior sequence (search, detail, cart). Core trade-off: richer System Prompt → higher accuracy but linear latency increase (measured on Vivo S20). Ablation experiments on fixed dataset quantified each prompt component's impact; final prompt retains category, rules, format, keywords; drops user profile and simplified variant.

LoRA Fine-Tuning Configuration

PEFT choice: LoRA (mainstream, resource-friendly). Key hyper-parameters (train.yaml):

model_name_or_path: /mnt/data/.cache/modelscope/models/Qwen/Qwen3-0.6B
finetuning_type: lora
lora_target: all
lora_rank: 8
lora_alpha: 16
lora_dropout: 0
num_train_epochs: 5
learning_rate: 5.0e-05
lr_scheduler_type: cosine
bf16: true
val_size: 0.05
eval_strategy: steps
eval_steps: 100
load_best_model_at_end: true
metric_for_best_model: eval_loss
greater_is_better: false

Practical tips: use CPU for non-GPU setup steps; feed eval results to AI for trend analysis & drill-down; script environment re-initialization (init_env.bash) to survive disconnects.

Model Pruning with ShortGPT

Goal: reduce size & latency without accuracy drop. Constraints: minimal pipeline disruption (reuse LLaMA-Factory + AliNNPrivate), simultaneous reduction in size/latency/memory, accuracy parity.

Evaluated mainstream methods (magnitude, Wanda, SparseGPT, ShortGPT, etc.); selected ShortGPT (Men et al., 2024, arXiv:2403.03853) using Block Influence (BI): BI = 1 − E[cos(h_in, h_out)] where h_in, h_out are layer input/output hidden states; cosine similarity near 1 indicates near-identity (pass-through). Compute BI on calibration set (average over samples & token positions), protect first/last layers, drop K layers with smallest BI.

Qwen3-0.6B has 28 transformer layers. BI distribution shows U-shape (high at ends, low in middle). Pruning experiments:

Drop 6 layers: on-device accuracy 79.8% (iOS) / 79.8% (Android).

Drop 8 layers: 73.7% (iOS) / 76.8% (Android) — speed gain insufficient to offset 3–6 pp loss.

Drop 10 layers: further degradation.

Production choice: drop 6 layers. Post-pruning requires healing fine-tuning (2 changes vs. initial LoRA: model_name_or_path points to pruned model; num_train_epochs reduced from 5 to 3 to avoid overfitting to pruning noise).

Backup explored: Minitron-style logit distillation (loss = (1−α)·CE + α·KL·T², α=0.9, T=1.0) via LLaMA-Factory source modification (enabled by AI coding), but underperformed healing LoRA.

Quantization: HQQ vs. AWQ

FP16 Qwen3-0.6B ≈ 1.2 GB. HQQ INT4 → ~300 MB (4× reduction). AWQ with custom calibration dictionary (replacing default English wiki) increased export time (<1 min → ~40 min) and artifact size (288 MB → 376 MB) but boosted on-device benchmark accuracy from 79.8% to 91.8%. AWQ adopted for production.

Inference Pipeline & Optimizations

On-device inference architecture: config-driven, observable, canary-enabled layers — Config Center (model URL, switches, user profile, fatigue, trigger rules), Pipeline Manager (download & pre-flight checks), Inference Pipeline (MNN SDK invocation & intent dispatch), Monitoring (remote dashboard + local ring-buffer logs).

Core flow: (1) config fetch (local cache first, async refresh on MD5 change); (2) model download (storage ≥1 GB free); (3) behavior recording → rule engine (e.g., ≥3 detail PVs) → prompt generation → inference request; (4) five gates (global switch, device allowlist, fatigue, model readiness, runtime health); (5) MNN inference → strip thinking tokens → keyword match → four intent outputs; (6) dispatch shopping_hesitate_trigger / shopping_scene_trigger events; (7) full-chain telemetry for funnel dashboards.

Performance Optimization Techniques

Prompt compression : shorten System Prompt within accuracy budget.

Prefix Cache : reuse prefill KV cache for repeated/static prompt prefixes.

Incremental Decoding : leverage KV cache across decode steps (standard).

Benchmark suite : 15 metrics across 4 tiers (P0: TTFT, decode throughput, peak RSS, model load time). Two test modes: (a) prompt-length buckets (128/256/512/1024 tokens from val set); (b) continuous stress test (stability, memory leak, thermal throttling). Automated reports enable cross-version comparison.

Business Results

A/B test (40% traffic, server-side bucketing, cloud LLM final coupon decision) shows sustained incremental GMV per 1 RMB investment. Recent week data: experiment group GMV consistently positive vs. control.

Hidden Cost: CDN Download Optimization

Model zip ≈ 280 MB; CDN cost non-trivial. Optimizations: on-demand download (only for eligible users), resume support, delta updates, cache reuse across versions.

Future Roadmap

Expand intent taxonomy to cover interactive non-buying users.

Push on-device accuracy from 80% → 90% via pruning + AWQ + quant_block=32 pipeline.

Deploy incremental inference widely to achieve <1 s end-to-end latency.

Build on-device agent with cross-session memory for habit-aware reasoning.