Taming LLMs with Deterministic Workflows: Dynamic Rule Engine for Asset Acceptance

Background: Why Not Traditional Rule Engines?

The asset acceptance scenario involves hundreds of approval orders per month, each requiring verification of 25-page PDF contracts, scanned documents, and attachments against dozens of expense types with distinct rules. Three core challenges make traditional rule engines impractical:

Many expense types, varied processes: Dozens of expense types each have unique acceptance flows and rules.

Rules are non-fixed and continuously adjusted: Business changes, risk control updates, and audit requirement shifts cause frequent rule additions, deletions, and modifications.

Large volume of unstructured documents for verification: Each order includes headers, multiple line items, contracts, invoices, and various attachments; manual point-by-point comparison is time-consuming.

The conventional approach of translating rules into code fails because:

Frequent rule changes: Every adjustment requires a full develop-test-release cycle with high response cost.

Rules operate on unstructured materials: Many rules require reading PDF contracts, scans, images, seals, and signatures — traditional programs cannot "understand" these.

Rules are expressed in natural language: Statements like "contract party must match acceptance order company" or "attachments must bear supplier seal" are semantic judgments, not numeric calculations — exactly what LLMs excel at.

The chosen approach: Let LLMs handle "reading materials and comparing against rules," while the engineering system handles "process orchestration, quality assurance, and traceability." Rules are configured as text and pushed down, no longer hard-coded.

Overall Architecture: Online Traffic Processing Flow

The system consists of two services. Key design points:

Async acceptance + callback delivery: A single LLM acceptance run parses multiple materials and executes multiple rules, taking minutes. The asset agent system returns "accepted" within seconds, executes in background, then callbacks the asset management system. Callback context is carried in the message, not stored server-side.

Idempotency and fallback: Idempotent batch numbers are constructed from "approval instance + finance node round." Re-signing, retries, and re-submissions never trigger duplicate execution. Submission failures synchronously record failed detail rows; query-side lazy timeout fallback ensures tasks never "hang."

One channel, three traffic types: Online, trial-run, and backtest traffic share the same submission and callback chain, distinguished only by traffic type + context. New traffic patterns reuse all infrastructure at zero cost. Different traffic types are fully isolated to prevent trial/backtest traffic from polluting online traffic.

Dual-channel image and text processing: The agent processes images and text via independent channels, reducing token consumption while maintaining recognition accuracy.

Quality System: Feedback, Trial-Run, Backtest, and Release Gates

Acceptance rules are essentially prompt text; changing rules changes online behavior. Since rules are non-fixed and adjust frequently, a lightweight "test-regress-release" mechanism is mandatory. The "standard answer" comes not from manual labeling but naturally from daily reviews .

User Feedback: Source of Standard Answers

Approvers can lightly annotate each AI judgment — accept/reject ; rejection requires a textual reason. The system records two snapshots at feedback time: the AI judgment at that moment and the effective rule version. Standard answers are derived automatically:

Accept → standard answer = original AI judgment

Reject → standard answer = opposite of AI judgment (pass ↔ reject)

Feedback data feeds two paths: as baseline for backtest comparison, and into trial-run to compute consistency rates between old and new versions.

Trial-Run and Backtest

Purpose: Before rule release, rehearse draft rules on real acceptance orders without affecting real online orders or approval flows . Acts as a debugging entry for business users to perceive draft version effects in real time.

Implementation: Admin selects historical/in-flight orders meeting criteria (time range, matching expense types) to initiate trial-run. System rebuilds acceptance context (header, details, attachments, contracts) in real time; both "draft version" and "current effective version" run once, results stored in independent side-channel tasks — no write-back to order status, no notification to approvers. Trial-run results support side-by-side old/new comparison, automatically compute judgment consistency rate , and give suggested pass/fail conclusion against user feedback.

Backtest dataset: Online traffic automatically deposits replay snapshots (acceptance context + rule version at that time) at each acceptance, forming a replayable historical sample pool. Combined with user feedback, samples are divided by "AI judgment × human standard answer" into four categories and stratified sampling is applied:

Stratified sampling purpose: Random sampling would let simple samples dominate, yielding unconvincing high scores. False positives/negatives are exactly what rule adjustments must watch — focus on whether new rules fix historical misjudgments or introduce new ones.

Backtest pass condition (dual thresholds) requires both :

Quantity threshold: Four categories of valid samples (with human standard answers) reach configured minimum, avoiding small-sample conclusions.

Quality threshold: Backtest result consistency rate with standard answers exceeds configured threshold (e.g., ≥90%).

Rule version release is gated by backtest status: only "backtest passed" or "trial-run substituted for backtest passed" allow regular release (version pointer switch takes effect immediately); other states (pending backtest/backtest failed) only allow forced release by privileged users with full audit trail. Any draft edit automatically resets backtest status — preventing old backtest conclusions from passing new rule versions.

Trial-run substituting backtest: Solves cold-start deadlock where "backtest pass required for release" but no historical samples exist yet. Privileged users manually review trial-run results (per-judgment, old vs new comparison, reasoning with material sources), confirm correctness, then manually set backtest status to passed — using human judgment on trial-run results to substitute automatic comparison against historical standard answers . Strictly limited to cold-start: permission control, full audit trail, draft edit resets backtest status; once sample pool is sufficient, rule adjustments return to standard backtest channel.

Three Traffic Types Data Isolation

Trial-run/backtest share the same submit-execute-callback chain as online traffic, premised on strict isolation — neither affecting online acceptance nor polluting backtest sample pool. Isolation enforced at four gates:

Thus trial-run/backtest fully reuse production-grade execution chain while remaining mutually isolated: no impact on online approval, no pollution of release-gate-dependent dataset. This forms a closed data flywheel loop:

Rule adjustments are no longer "seat-of-the-pants releases"; every modification is validated against historical standard answers.

Agent Design: How Deterministic Workflow Tames LLM

This chapter details the technical foundation enabling "rules as configuration." All technical details come from actual code and measured data.

Overall Approach: Let LLM Do Only Two Things It Excels At

No conversational multi-agent architecture; instead a deterministic workflow orchestration pipeline design, splitting one acceptance into fixed stages:

LLM appears only in two green stages: extracting facts from materials, and comparing facts against rules for judgment. All other stages — file parsing, seal detection, numeric verification, conclusion validation, timeout fallback — are handled by deterministic code.

Benefits: predictable behavior, testable stages, localizable issues. For production systems, this matters more than "looks intelligent." Why not autonomous looping agents? Because this task is a directional workflow, not exploratory; stages, I/O, success criteria are fixed at design time. Loops introduce unpredictable step counts and behaviors, contrary to "acceptance conclusions must be predictable and regressable." Effort goes into stage decomposition, compressing judgment inputs, and voting on critical judgments — not letting the model decide next steps. Consistent with frontend trade-offs: deterministic TypeScript-expressible flows shouldn't be handed to runtime adaptation.

Layered Architecture and Processing Flow

Routing layer          Thin routing: read raw request / context logging / error code mapping, zero business
 └─ Async orchestration  Acceptance validation / backpressure slots / background tasks / failure retry
    └─ Execution kernel  Concurrency semaphore / budget / material prep / phased scheduling / timeout fallback
       ├─ Parsing module  Pure functions: input parsing + amount precision + contract compatibility
       ├─ Mapping module  Pure functions: rule translation / prompt templates / result mapping
       └─ Reuse existing engines:
            ├─ Audit engine  Concurrent execution, multi-value result protocol, multi-model voting
            └─ Material processing  Download, categorize, extract, render, element detection

Zero-business routing: Only four duties — read raw body, four-level validation funnel, backpressure placeholder, background registration. All business orchestration pushed down to async.ts; error code mapping unified at entry, gateway unaware of internal structure.

Kernel single responsibility: Execution kernel only knows tasks, not requests — manages only concurrency semaphore, budget, material prep, phased scheduling. Acceptance and callback lifted to async orchestration layer; same kernel reusable by sync, async, retry three entry types.

Rules decoupled from engines: Parsing and mapping modules are pure functions — input parsing, amount precision, contract compatibility, rule translation, prompt templates, result mapping; no I/O, no side effects. Any business rule change converges to unit-testable function changes; underlying audit and material capabilities abstracted as engine services, multi-model voting, element detection implemented once, upper orchestration iterations don't touch engine code.

Request lifecycle:

The first row "input validation + callback block validation + backpressure placeholder" deserves expansion — the acceptance entry is the highest design-density segment, a multi-level serial fail-fast funnel where any level failure immediately rejects, no background task registration, no callback. Example failure codes and upstream actions:

Benefits: (1) Error code = retry strategy: Multiple failure codes let upstream code distinguish "fix won't help" from "retry later succeeds," enabling different upstream actions per code. (2) Accept only after all acceptance validations pass: Upfront rejection cost far lower than letting task run full timeout then lazy-timeout invalidate.

Material Preparation Pipeline

Core principle: "Each material downloaded once, each takes its most suitable form." Implementation highlights:

Single buffer reused end-to-end: Download is heavy I/O; same file downloaded once, shared by page extraction, trigger-page rendering, seal detection.

PDF type intelligent downgrade: Auto-routed by per-page character threshold — electronic contracts (text-type) zero-cost direct text-layer extraction; scans (image-type) go through rendering + OCR. Both paths output unified structure.

On-demand rendering: Only pages with seal/signature visual rules are rendered; pure text rules don't pay for rendering.

OCR and Vision each do their job: OCR only does "text keyword location" — identifies pages possibly involving seals/signatures, backfills scanned text into text slots. Vision only does what OCR cannot — "graphic judgment": does this page have a seal? handwritten signature? Not using weakest means (pure OCR for seal judgment) for critical judgments, nor letting vision model chew entire text for trigger words. Correct tool division is the foundation of this hybrid approach.

Dual-Artifact Model: Extraction and Judgment Separation

Each upstream rule translates into two engine artifacts :

Example rule: "Acceptance quantity must match contract agreed quantity." Two artifacts' real outputs:

Extract artifact (facts, verbatim extraction with traceability):

{
  "needed_fields": ["合同约定数量", "验收单数量"],
  "extracted": [
    { "field": "合同约定数量", "value": "20", "source_file": "合同.pdf", "source_page": 3, "source_method": "ocr" },
    { "field": "验收单数量", "value": "20", "source_file": null, "source_page": null, "source_method": "extraContext" }
  ],
  "judgment": "NULL",
  "reason": "合同数量取自 p3;验收数量取自单据结构化字段"
}

Aggregate artifact (structured comparison items, no free text):

{
  "checkResult": 1,
  "comparisons": [{
    "leftName": "合同约定数量",
    "leftValue": "20",
    "leftSource": "《合同.pdf》p3",
    "rightName": "验收单数量",
    "rightValue": "20",
    "rightSource": "补充上下文",
    "relation": "一致"
  }]
}

Single responsibility yields four benefits:

Auditable: Judgment reasoning must cite specific values and sources (no vague "all match"); every value traceable to page number.

Cost control: Judgment stage only sees fact summary, doesn't re-read raw materials; token and latency controllable.

Short-circuitable: Deterministic judgments (e.g., seal detection) can skip LLM entirely in extraction stage (see 4.7).

Verifiable: Both segments output JSON; model output four-value → rule judgment two-value mapping can be pure-function validated.

Multi-Model Voting: Pressing Single-Point Misjudgment onto Critical Judgments

Acceptance is AND-semantics: one critical fact misjudged flips the whole order conclusion. LLMs exhibit noise on same input: blurry scans, e-seals not traditional round red seals, cross-page seam seals — all cause single model to "misread." Especially visual judgment (seal present/absent) accuracy is the floor of the whole solution: single page misjudging "has seal" as "no seal" causes a fully compliant bid to be entirely rejected; single-point misjudgment cost too high, must hedge with mechanism.

Solution: Majority vote — each critical judgment not given to single model, but primary model + a group of voting models answer concurrently, take majority answer. Voting scope is bounded — only pay for voting on critical judgments :

Voting mechanism plays irreplaceable role in two places:

Reduce output drift: Break one long question into multiple sub-questions (corresponding to multiple functions), each function returns value fixed in small legal set (yes/no), model answers only on small closed set, then majority vote converges — stability of repeated same-input calls significantly higher than letting model freely generate long answer.

Seal confirmation (see 4.7 Stage 3): Vision binary classification "is seal" runs three-model vote; after suppressing "single-model visual seal judgment" noise, first two deterministic filtering stages can confidently judge non-detected pages as "no seal."

Three-Stage Seal Detection Pipeline: Deterministic First, LLM Fallback

"Attachments must bear supplier seal" is a high-frequency rule. Throwing entire contract to Vision model asking "sealed?" is expensive and unstable. Solution: three-stage pipeline, first two stages zero LLM cost:

Every threshold comes from per-page real measurement on real contracts (validated on a 25-page text-type contract):

Anti-Hallucination Combo

Finance scenarios tolerate no "looks right." Hallucination-prone points sealed one by one with engineering means:

Each measure in table has corresponding code landing, not relying on model "self-awareness." Two trickiest — answer value domain convergence, amount from message to conclusion — each explained with a diagram.

Four-value closed-set contract: Converging model answers into deterministic value domain. All LLM calls go through unified ask engine; answer value domain nailed at type level:

export type CheckVal = 'true' | 'false' | 'NULL' | 'N/A'
// true=pass; false=fail; NULL=no judgment needed; N/A=unable to judge (e.g., .zip/.pptx unparsable materials)

Closed-set enforced doubly: prompt side declares "only return yes, no, NULL, or N/A"; code side forces any out-of-bounds answer to fallback to last legal value — model output whatever, cannot produce conclusion outside four values. Full convergence path from raw answer to final conclusion shown below; fallback direction always conservative: worst case "unable to judge, escalate to human" rather than fabricate true/false.

Amount pipeline: From raw message to final conclusion, every link distrusts model. BigDecimal literal pre-extraction, numeric two-way arbitration, conclusion derivation reconciliation chained into a pipeline — model only appears at stage ③, rest all deterministic code:

Explainable Structured Output

LLM doesn't write free-text judgment reasons; instead outputs structured comparison items (left value/right value/each source/relation), assembled by code via fixed templates into rich text. When field extraction fails, falls back to missing-material form:

{ "leftName": "合同约定数量", "relation": "缺失", "expectedSource": "采购合同正文" }

Assembled as "Contract agreed quantity missing (should appear in purchase contract body), this part cannot be judged" — not letting LLM freestyle vague "upon verification unable to obtain relevant information." Every judgment presented to approver carries rule original text and material sources (which contract, which page), enabling one-glance manual review and naturally supporting feedback annotation.

Above is missing-material form; normal form is comparable items — besides left/right values and relation, each item must carry each source ("filename"+page, or "extra context") and ruleRef (verbatim quote of one rule from sub-rule list, for code to attach reason back to rule). Example "party names must match": aggregate segment output comparison item:

{
  "leftName": "合同甲方主体",
  "leftValue": "XXX有限公司",
  "leftSource": "《xxx协议.pdf》p3",
  "rightName": "验收单公司主体",
  "rightValue": "XXX有限公司",
  "rightSource": "补充上下文",
  "relation": "一致",
  "ruleRef": "1、各方主体名称应一致"
}

Relation is closed-set; rendering forms four types, each with dedicated template (table shows pure text stripped of <val> tags):

Program assembles via fixed templates into tagged statements, with frontend-agreed display rules; every key part in result assembled by code, LLM never sees HTML. Programmatic assembly guarantees result consistency. Example:

<judge verdict="通过" rule="各方****一致性"></judge><br/>来源： <sources><a href="https://xxxx.pdf">《合同A.pdf》</a>、<a href="https://xxxx.jpeg">《图片B.jpeg》</a></sources><br/>理由： 经比对，验收界面公司主体()与合同、补充协议、订单附件及验收附件中的我方主体名称完全一致;验收界面供应商主体()与上述各文件中的对方主体名称完全一致，满足主体一致性要求 因此整体判断为通过<br/><b>因此整体判断为通过。</b><br/>置信度： 85%

Production-Grade Engineering Guardrails and Lessons Learned

Backpressure and budget layered: In-flight task upper limit threshold; when full, reject immediately — silently accepting a doomed-to-timeout task is worse than letting upstream immediately record failure, avoiding avalanche.

Failure explicit: Material download failure injected into both prompt and judgment reason, explicitly stating "this judgment based on incomplete materials"; silent degradation prohibited.

Context engineering: Text budget evenly divided by file count, preventing main contract from squeezing out related contracts; numeric literals retained verbatim before JSON parsing.

Prompt Cache stable prefix breakpoint: System message split into two parts — stable template (role definition + output contract + anti-injection statement) marked with cache_control: ephemeral breakpoint; rule instructions, materials, images all placed after. Prompt Cache is prefix-match; this organization lets stable template hit cache across rules, requests, voting models. Cache hit occurs at remote LLM gateway: gateway tokenizes pre-breakpoint content, computes hash, looks up matching cache block in backend KV store; hit → prefill stage reads KV directly from cache, no per-token recomputation — saves real compute and first-token latency, not local response-level reuse. User segment never gets breakpoint — it varies per rule, participating in breakpoint would make cache never hit.

Config-driven: Budget/backpressure/timeout/retry all via config files; parameter tuning without deployment.

Full-chain observability: Entry logs identity (including full input, issues replayable on-site); acceptance, execution, callback three segments linked by same ID, one grep locates full chain; acceptance success and backpressure rejection logs both carry pending watermark, capacity retrospection without pre-buried monitoring.

Sensitive info desensitization: Contracts and order materials pass through desensitization before entering LLM — phone, email, printer info directly erased; business metrics (GMV/ROI/UV/PV/DAU) and other sensitive words replaced by placeholders per config table. Desensitization happens at prompt assembly stage; LLM always sees desensitized version; judgment-needed fields (entity names, amounts, dates) unaffected. Sensitive word list config-driven; new entries added without deployment.

Lessons learned from real pain:

Effect Showcase

Actual system screenshots along "approval review → rule config → version management → trial-run/backtest" mainline.

Acceptance Approval Page — AI Approval Result Detail Drawer

After acceptance order reaches finance approval node, AI judgment results surface synchronously. Click to open "AI Approval Result Detail" drawer:

Rule judgment summary: Aggregates rule counts by pass/fail; approver grasps whole order at a glance, returns composite judgment: pass/fail/needs manual review.

Rule judgment detail: Per-rule shows rule ID, AI judgment result, rule original text and judgment basis; data values in basis (expense type, amount, contract ID, etc.) rendered as hyperlinks, click to view data source , facilitating item-by-item verification.

Sources: Click file links to open analyzed materials.

Feedback: Each judgment bottom provides "correct/incorrect" buttons; approver annotations precipitate as standard answers, feeding back into trial-run/backtest.

AI Acceptance Rule Config Page and Version Management

Rule config: List view shows all rule dimensions: rule ID, name, expense type, level (blocking/major/minor), status (enabled/disabled), creator, creation time; top supports combined filtering by expense type, creator, status for quick locate; actions column provides detail, edit, version management, disable, delete.

Version management: Each rule supports multiple versions: page lists all versions (draft, v1, …) and whether effective ("in effect" / -). Every version supports trial-run and backtest ; draft version additionally provides edit and backtest entry; release button appears only after backtest passes ; rule release gated by backtest status.

Trial-Run and Backtest Results

Records historical trial-run results; shows draft vs current version conclusion differences; if human feedback (gold standard) exists, displays gold standard results.

Backtest result list, consistency rate with standard answers, backtest status.

User Feedback Entry

For each rule's LLM judgment, user can click "correct" or "incorrect" at judgment bottom: "correct" accepts AI judgment; "incorrect" pops error feedback modal requiring reason (mandatory on reject, ensuring feedback information).

Feedback data has two destinations: (1) Derive standard answers, feed trial-run/backtest gates (see 3.1); (2) Error reason text itself — program uses cosine similarity to cluster different feedback into different Semantic Clusters , facilitating extraction of representative data from each cluster to generate few-shot examples, letting model learn from same-type historical error patterns, making rule iterations increasingly accurate.

Data Structure Design

Core tables supporting "trial-run/backtest/feedback" — five tables, one main line: every judgment hangs on "rule version," making history traceable and standard answers derivable.

Design rationale:

Three traffic types share replay snapshot table , logically isolated by traffic type + batch number ; no separate tables for trial-run/backtest; read side filters by traffic type and batch prefix — direct storage-layer mapping of "one channel three traffic types," new traffic modes zero table-creation cost.

Rule version number runs through result, snapshot, feedback three tables; every historical judgment can answer "which rule version was used then" — without this foreign key, "feedback derives standard answer" and "draft edit auto-resets backtest status" impossible.

Standard answer fixed at feedback write, set as gold set ; backtest/trial-run only need per-item compare model result vs gold to compute consistency rate, zero logic at query time — one-time derivation cost at write, high-frequency comparison made zero-logic.

Summary

Facing "many expense types, non-fixed rules, large verification volume" acceptance scenario, this solution answers with:

Rule configuration: Rules delivered as text, semantic routing; change rules without changing code.

Judgment AI-ified: LLM handles material understanding and rule comparison; deterministic workflow ensures process controllability.

Quality engineered: Trial-run rehearsal, backtest gates, feedback standard answers — every rule adjustment verifiable and regressable.

Human-machine collaboration: AI provides judgment and basis; human makes final decision and feeds back standard answers, forming data flywheel.

Monthly near-thousand acceptance verifications transformed from pure manual point-by-point to "AI pre-judgment + human review focus." Future: continue expanding expense type coverage and feedback data accumulation, making rule iterations faster and more stable.