AI-Driven Revamp of Banking Security Test Demand Assessment

1. Special Requirements and Evolution of Banking Security Testing

The banking sector’s strict security obligations, reinforced by laws such as the Cybersecurity Law, Data Security Law, and Personal Information Protection Law, make security testing a core SDLC activity. Traditional security demand assessment relied on manual experience, leading to inconsistent quality and knowledge loss.

The assessment process has evolved through three stages:

Experience‑driven (pre‑2015): Analysts manually interpreted requirements, causing high variance and knowledge fragmentation.

Rule‑driven (2015‑2023): A rule‑based checklist standardized evaluation but incurred high maintenance costs and still suffered from poor data entry quality.

Intelligent‑driven (2024 onward): Large language models (LLMs) enable automatic requirement parsing, risk detection, and test‑case generation, turning tacit expertise into reusable knowledge assets.

2. Pain Points of the Traditional Process

The pre‑AI workflow required developers to fill demand management systems and a separate security assessment form, leading to repeated data entry, inconsistent information, and low‑quality assessment tables. Four core pain points were identified:

Duplicate data entry across systems, reducing efficiency.

Inconsistent assessment‑form quality due to varying security awareness.

Test‑case selection heavily dependent on individual experience.

High cost of maintaining and updating rule sets.

3. Why AI Instead of Rule Optimization

Initial attempts to refine the rule engine revealed that rule‑maintenance cost grows with business complexity. Banking scenarios are diverse, and fine‑grained rules would require extensive upfront research and constant updates.

Rapid advances in LLMs, especially models with strong Chinese comprehension and code analysis (e.g., DeepSeek), offered semantic understanding and knowledge‑retrieval capabilities. Retrieval‑Augmented Generation (RAG) allowed the system to consult an enterprise security knowledge base while mitigating hallucinations.

4. Technical Architecture and Workflow

The solution reuses existing demand‑management and test‑management systems, adding a low‑code platform (Dify) that orchestrates AI calls to the DeepSeek model via new APIs.

Infrastructure layer: Linux servers, databases, and message queues.

AI capability layer: Dify visual workflow engine + DeepSeek LLM for multi‑step reasoning and knowledge‑base retrieval.

Application layer: Vue front‑end for user interaction; Django back‑end exposing APIs that pull demand data, invoke AI evaluation, and return results.

The end‑to‑end workflow consists of:

Demand synchronization: pull pending demands and pre‑fill basic fields.

Demand confirmation: developers verify or supplement details.

AI evaluation: the LLM retrieves relevant knowledge and generates a security assessment.

Human review: security experts validate AI output.

Security testing: test teams execute tests based on the approved assessment.

5. Knowledge‑Base Construction

Three core knowledge bases were built:

Security standards library: Regulatory documents (e.g., Financial Industry Cybersecurity Guidelines) and internal coding standards.

Historical test‑case library: Categorized test cases for internet‑facing, intranet, and API systems, stored as vectors for similarity‑based recommendation.

Vulnerability case library: Documented production vulnerabilities, severity levels, and remediation steps.

6. Results and Lessons Learned

During trial operation, the system achieved:

Efficiency gains: Data‑entry effort reduced by ~40‑50%; average assessment time dropped from ~1 hour to 0.5‑0.6 hour, nearly doubling throughput.

Quality improvements: AI assessments matched senior tester judgments >80% of the time; consistency across testers increased, especially for sensitive‑data scenarios.

Key experiences:

Human‑AI collaboration is essential; AI assists but does not replace expert judgment.

Knowledge‑base quality directly impacts AI output accuracy; expert curation is required.

Gradual rollout, starting with non‑core systems and retaining legacy processes as fallback, eases adoption and builds trust.

7. Future Outlook

The next phase will extend AI assistance to test execution and vulnerability verification, creating a full‑link intelligent system covering assessment, testing, and validation. Planned capabilities include automatic test‑script generation, AI‑guided vulnerability analysis, and predictive risk models that flag high‑risk features during requirement gathering.

Ultimately, the goal is to shift security left into the requirement stage and embed AI‑driven safety awareness throughout development, fostering a culture where “everyone understands security” and risks are mitigated before they materialize.