How AI Powers Financial Form Automation and Insurance Q&A: Open‑Source Solutions

Background

Artificial‑intelligence techniques such as computer vision, speech and natural‑language processing are increasingly applied in the financial industry to lower operational and risk costs and to improve customer experience. Typical use cases include OCR‑based form extraction and NLP‑driven intelligent question‑answering.

Form Recognition Scenario

The task is to extract key‑value pairs from a variety of document‑style forms (e.g., property certificates, business licences, personal information sheets, invoices). These forms appear in banking, securities and corporate finance and have high commercial value.

Challenges

Large variety of form layouts requires a highly compatible solution.

Traditional single‑modal OCR pipelines generalise poorly and need massive labeled datasets.

Solution Architecture

The pipeline consists of two stages:

OCR stage : PaddleOCR PP‑OCRv2 is used. It contains a lightweight text‑detection head and a text‑recognition head. Both heads are first evaluated on the multilingual XFUND dataset (a public benchmark containing diverse form types) and then fine‑tuned on the same dataset. An additional fine‑tuning step with real‑world annotated images further improves recognition accuracy.

Document Visual Question‑Answering (Doc‑VQA) stage : PaddleNLP’s LayoutXLM model is employed. LayoutXLM supports multimodal Semantic Entity Recognition (SER) and Relation Extraction (RE). The model is pre‑trained on the Chinese portion of XFUND and subsequently fine‑tuned for SER and RE tasks.

Model Optimization and Results

Text detection : The lightweight detection head of PP‑OCRv2 is first benchmarked on XFUND, then fine‑tuned on the same data, yielding higher detection precision.

Text recognition : Three configurations are compared – (1) the base PP‑OCRv2 model, (2) fine‑tuned on XFUND, and (3) fine‑tuned on XFUND + real‑world data. The third configuration achieves the best character‑level accuracy.

Doc‑VQA (LayoutXLM) : After pre‑training on the Chinese XFUND data, LayoutXLM is fine‑tuned for SER and RE. Evaluation on the XFUND test split shows strong F1 scores for both tasks.

Insurance Smart Q&A Scenario

In insurance, 60‑70 % of user inquiries are repetitive, making manual handling inefficient. An intelligent Q&A system can understand user intent and return precise answers without requiring deep domain expertise.

Challenges

High domain specificity makes semantic modeling difficult.

Annotated question‑answer pairs are scarce and expensive to obtain.

Solution Architecture

The approach combines RocketQA (a dense retrieval model) with SimCSE (a contrastive sentence encoder) to build a retrieval‑augmented QA system. The pipeline includes:

Selection of a base retrieval network (RocketQA) and a sentence encoder (SimCSE).

Strategy enhancements such as synonym replacement, Word‑Replacement (WR) and R‑DROP regularisation.

Hyper‑parameter tuning (learning rate, batch size, number of negative samples).

Fine‑tuning on a small, domain‑specific QA set.

Model Optimization and Results

Applying synonym replacement, WR and R‑DROP improves the retrieval quality, achieving 96.433 % Recall@10 on the internal test set.

Deployment

Both the OCR/Doc‑VQA models and the retrieval‑augmented QA model are served with Paddle Serving. Measured latencies are:

Vector retrieval: 7 ms

Model inference: 12.7 ms

Tools and Resources

PaddleOCR provides a comprehensive OCR model library, data‑synthesis utilities and semi‑automatic annotation tools, enabling rapid development of industrial‑grade OCR pipelines.

PaddleNLP offers a full‑stack NLP toolkit with pre‑trained models, data‑processing pipelines, model optimisation utilities and deployment support, facilitating retrieval‑augmented Q&A systems without large labeled corpora.

Source code and tutorials are publicly available:

https://github.com/PaddlePaddle/PaddleOCR/tree/dygraph/applications

https://github.com/PaddlePaddle/PaddleNLP/tree/develop/applications

https://github.com/PaddlePaddle/RocketQA