Systematic Evaluation of Interaction Capabilities for Large‑Model Applications

1. Testing Challenges for Large‑Model Applications

Large‑model application systems differ fundamentally from traditional software, introducing five testing challenges: (1) random customer behavior, (2) diverse input expressions, (3) non‑unique outputs, (4) model instability caused by temperature and random seeds, and (5) exponential growth of test cases due to numerous input dimensions. Conventional techniques such as equivalence partitioning and boundary analysis cannot adequately address these issues, necessitating a new systematic testing framework.

2. Interaction Capability Testing Framework

To address the challenges, the bank focuses on four foundational capabilities—basic dialogue, information extraction, intent recognition, and process decision. Each capability is broken down into specific dimensions:

Basic Dialogue : single‑turn accuracy, multi‑turn context coherence, noise tolerance and robustness.

Information Extraction : accuracy, completeness, robustness, complex‑structure extraction, and multimodal extraction.

Intent Recognition : clear intent classification, handling of complex structures, and identification of implicit/deep intents.

Process Decision : adherence to fixed processes, branch decision accuracy, and closed‑loop result output.

The framework is visualised in Figure 1 (see image below).

Special scenario expansion uses a CRAFT‑based prompt template (Figure 2) to generate diverse business dialogues automatically.

3. Automated Semantic Verification

The bank implements an automated verification method centred on cosine‑vector similarity (Figure 3). The workflow consists of:

Automatic feeding: test scripts read standardized input phrases and feed them to the target model.

Output collection: the system captures the model’s actual responses.

Semantic vectorisation and similarity calculation: both expected and actual outputs are encoded into dense vectors using a pretrained word‑embedding model, and cosine similarity is computed (range [-1, 1]).

Result judgement: a similarity threshold determines pass/fail, with failures recorded as defects.

The method offers three advantages: full automation and high efficiency, semantic‑level assessment that tolerates surface wording differences, and a quantifiable quality standard via similarity thresholds.

4. Pilot Validation and Results

A pilot in the bank validated the framework’s practical value. Key benefits observed were:

Comprehensive scenario coverage, eliminating blind spots.

Continuous expansion of the test‑case library, enhancing depth and completeness.

Effective defect detection, especially for complex logic missed by traditional cases.

The pilot also highlighted current quality gaps—multi‑turn intent handling and dialogue stability require improvement.

Automation yielded an 89% efficiency gain: a test case that took ~3 minutes manually required only ~20 seconds with the automated tool.

5. Conclusion

The study systematically analyses interaction‑capability testing for large‑model applications, builds a layered framework, and introduces an automated semantic verification approach that delivers measurable efficiency and defect‑detection improvements. Nonetheless, the field lacks unified standards, leading to fragmented practices. Future work should involve industry collaboration to define common processes, methods, and evaluation metrics, thereby advancing reliable AI‑native deployments across sectors.