Financial Large Language Models: Architecture Shifts, Engineering Lessons, and Cutting‑Edge Agent Strategies

Background and trends – In finance, three hard constraints—regulatory traceability, private‑deployment security, and strict data‑business integration—define the limits of large‑model adoption. The team concluded that generic models cannot be applied directly; instead, the IT architecture must be rebuilt around atomic business capabilities (Skills), plug‑in‑style LLMs, and a model‑friendly data layer (AIDB).

Scenario sharing – Real‑world cases illustrate value: (1) a client‑onboarding flow requiring >200 documents is transformed by an LLM that converts natural‑language intent into system operation sequences, akin to moving from a map to a navigation guide; (2) in wealth‑management, an insurance claim scenario exposed a rule (“six‑month holding period”) that traditional search could not close the loop; (3) an investment‑banking prospectus of 1,300 pages rendered conventional RAG ineffective, highlighting the need for long‑document retrieval.

Engineering practice – Four challenges of long‑document retrieval are identified: massive chunk explosion, high‑frequency term ambiguity, table‑split across chunks, and the necessity to scan entire documents for numeric queries. The solution, PageIndex , parses document headings offline and maps chapter names to page ranges, compressing a 300‑page search space to three pages before fine‑grained chunk retrieval. Combining PageIndex with Agentic RAG, BM25, and vector search yields >95% chunk‑recall; however, in 2023 the team removed vector search because exact‑match needs in finance favor BM25. Model selection lessons: using Qwen3‑32B required 530 rules and 4,300 lines of code, leading to high staff turnover; switching to Qwen3‑235B on H800/H20 GPUs (≈¥600k investment) cut rules dramatically and improved accuracy by ~45 percentage points. Prompt engineering reduced a 24 k‑token prompt to <3 k tokens by dynamically injecting 180 financial indicators and reusing chapter headings and table headers.

Agent exploration – Financial agents must go beyond chatbots to become executors that read files, call APIs, write results, and ask humans when needed. Success requires (a) model‑level task decomposition, planning, and self‑evaluation; (b) stable function‑call support; (c) downstream tools expressed as atomic Skills with MCP protocol for retryable calls; and (d) a robust data layer (AIDB) with knowledge slicing and business‑oriented API descriptions. The OpenClaw case reveals four shortcomings: vague permission boundaries, insufficient audit granularity, uncontrolled plugins, and hallucination safety gaps. In advisory agents, each Skill must specify required materials, data sources, and permission levels; common missing items include outdated risk assessments and non‑real‑time product status. API redesigns replace opaque descriptions with clear, time‑bound, business‑semantic specifications.

Future outlook – The long‑term vision shifts developers from code generators to business auditors, emphasizing the construction of data foundations, interface standards, and knowledge bases rather than training proprietary models. Deliverables should be “Lego‑style” Skills—complete, composable units—while AI agents act as the execution layer. The ability to encode business constraints into LLM context will become increasingly scarce and valuable.