From Beginner to Expert: AI‑Driven Testing of a Telecom Settlement System – Full‑Process Guide

Overview of Traditional Testing Pain Points

Before AI adoption, most enterprise‑grade systems rely on manual test case creation (20‑30 cases per person per day), manual maintenance after code changes, impact analysis based on experience (30% change‑omission rate), and manual log inspection (average 30 minutes per case). Reports are static and lack actionable insights.

The China Unicom settlement system adds extra challenges: hundreds of complex billing rules (domestic roaming, international roaming, value‑added services) with inter‑dependent logic, extremely high data‑consistency requirements (errors can cause tens of millions of loss), and a multi‑language stack (Java, Go, C).

AI‑Driven Testing Paradigm Shift

From passive response to proactive prediction

From full‑suite execution to precise, targeted testing

From manual maintenance to AI‑self‑healing

From static result reports to root‑cause analysis

Typical Pain‑Point Scenario

Scenario: The operator adjusts the "International Roaming Night Discount" rule, changing the night‑time window from 23:00‑07:00 to 00:00‑06:00.

Traditional workflow

Testers receive the change notice.

Manually locate all ~320 test cases involving night discounts.

Manually edit time parameters and expected results.

Re‑execute the suite – about three person‑days of effort.

Miss edge cases (e.g., cross‑timezone calls) → production issue.

AI‑driven workflow

Git hook triggers AI analysis.

Within 5 seconds AI identifies every affected test point.

AI automatically generates updated test cases.

AI also creates additional edge‑case tests for cross‑timezone scenarios.

Learning Objectives (2‑day course)

Design an AI testing architecture independently.

Write efficient testing prompts.

Implement Git‑based smart change analysis.

Build a multi‑language AI testing solution.

Design cost‑controlled AI invocation strategies.

AI Testing Automation Architecture

The solution is divided into four layers, each responsible for a specific function.

Code Layer: Git change analysis → AI impact scope identification.

Test‑Case Layer: AI generates or updates test cases.

Execution Layer: Adaptive assertions with self‑healing.

Analysis Layer: AI root‑cause analysis + quality heatmap.

Code Layer – Pseudocode

class CodeAnalysisLayer:
    def analyze_change(self, commit_id):
        diff = self.git.get_diff(commit_id)
        changed_files = self.extract_files(diff)
        changed_methods = self.extract_methods(diff)
        changed_fields = self.extract_fields(diff)
        prompt = build_impact_prompt(
            diff=diff,
            changed_methods=changed_methods,
            context=self.get_code_context(changed_files)
        )
        impact = self.llm.analyze(prompt)
        return ImpactReport(
            affected_apis=impact.apis,
            affected_db_tables=impact.tables,
            affected_business_rules=impact.rules,
            risk_level=impact.risk
        )

Sample JSON output:

{
    "affected_apis": ["/api/billing/calculate", "/api/invoice/generate"],
    "affected_db_tables": ["t_settlement_detail", "t_discount_rule"],
    "affected_business_rules": ["夜间优惠规则", "节假日双倍积分规则"],
    "risk_level": "HIGH",
    "recommended_tests": ["边界值测试", "跨时段场景测试", "并发测试"]
}

Test‑Case Layer – Generation

class TestCaseLayer:
    def generate_test_cases(self, impact_report, business_rules):
        test_cases = []
        for api in impact_report.affected_apis:
            api_spec = self.get_api_spec(api)
            prompt = build_testgen_prompt(
                api_spec=api_spec,
                business_rules=business_rules,
                test_types=["正常", "边界", "异常", "并发"]
            )
            cases = self.llm.generate(prompt)
            test_cases.extend(cases)
        return test_cases

Execution Layer – Self‑Healing

class ExecutionLayer:
    def execute_with_self_healing(self, test_case):
        try:
            result = self.run_test(test_case)
            self.assert_result(result, test_case.expected)
        except AssertionError as e:
            healed = self.self_healing_engine.heal(
                test_case=test_case,
                actual_result=result,
                error=e
            )
            if healed.success:
                self.update_assertion(test_case, healed.new_expected)
                return self.run_test(test_case)
            else:
                raise

Analysis Layer – Heatmap

class AnalysisLayer:
    def analyze_failures(self, failures):
        clusters = self.cluster_failures(failures)
        for cluster in clusters:
            prompt = build_root_cause_prompt(
                failure_logs=cluster.logs,
                code_context=self.get_code_context(cluster.location),
                environment=cluster.env
            )
            cluster.root_cause = self.llm.analyze(prompt)
        heatmap = self.generate_heatmap(clusters)
        return AnalysisReport(clusters=clusters, heatmap=heatmap)

Quality heatmap: visualizes problem density; hotter colors indicate more severe or frequent issues, helping quickly locate hot spots.

Practical Steps to Sketch Your Own AI Testing Architecture

Identify test objects (frontend, API, database, message queue, etc.).

Determine AI capability points (impact analysis, test generation, data construction, assertion update, failure root‑cause, report generation).

Design data flow: how changes trigger AI, how results are consumed.

Evaluate feasibility: technical difficulty, cost, expected benefit.

Git Integration and Change Impact Analysis

Key commands to extract diffs:

git diff HEAD~1 HEAD
git diff HEAD^ HEAD
git diff HEAD~1..HEAD

Python parser extracts added/removed lines, modified files, and dependency changes across Java, Python, Go, and C files.

class GitAnalyzer:
    def get_diff(self, commit_hash):
        cmd = f"git show {commit_hash}"
        result = subprocess.run(
            cmd, shell=True, capture_output=True, text=True,
            encoding='utf-8', errors='ignore'
        )
        return result.stdout
    def parse_diff(self, diff_text):
        changes = {'added_files': [], 'modified_files': [], 'deleted_files': [], 'code_changes': []}
        current_file = None
        for line in diff_text.split('
'):
            if line.startswith('diff --git'):
                match = re.search(r'b/(.+)$', line)
                if match:
                    current_file = match.group(1)
                    changes['modified_files'].append(current_file)
            elif line.startswith('+') and not line.startswith('+++'):
                changes['code_changes'].append({
                    'file': current_file,
                    'type': 'added',
                    'content': line[1:],
                    'line_number': self._extract_line_number(line)
                })
            elif line.startswith('-') and not line.startswith('---'):
                changes['code_changes'].append({
                    'file': current_file,
                    'type': 'deleted',
                    'content': line[1:],
                    'line_number': self._extract_line_number(line)
                })
        return changes

Prompt Engineering for LLM Impact Analysis

Principles:

Provide sufficient context (code diff, surrounding files, business rules).

Specify explicit JSON output format.

Limit analysis scope to avoid model drift.

Include few‑shot examples for higher accuracy.

Full prompt template (Python example):

def build_impact_analysis_prompt(diff, code_context, business_rules):
    return f"""
你是一位资深的测试架构师，擅长分析代码变更对系统的影响。
## 任务
分析以下代码变更，输出影响分析报告。
## 变更代码 (Git Diff)
{diff}
## 代码上下文
{code_context}
## 业务规则上下文
{business_rules}
## 输出格式 (JSON)
{{
    "change_intent": "",
    "affected_components": {{
        "frontend": [],
        "backend": [],
        "database": [],
        "business_rules": []
    }},
    "test_recommendations": {{
        "regression_tests": [],
        "new_tests": [],
        "edge_cases": []
    }},
    "risk_level": "",
    "risk_reason": ""
}}
## 分析要求
1. 变更意图
2. 影响范围（调用链、数据流、UI）
3. 具体可执行的测试建议
4. 风险评估（业务敏感度、变更复杂度、影响广度）
请开始分析：
"""

Case Study: Java Settlement Method Change

Original method calculates roaming fee at 10 cents/second, VIP 20% discount, night discount 30% (23:00‑07:00).

public class SettlementCalculator {
    public long calculateRoamingFee(CDR cdr, int userType) {
        long baseFee = cdr.getDuration() * 10; // 10分/秒
        if (userType == 2) {
            baseFee = baseFee * 8 / 10; // VIP 8折
        }
        if (isNightTime(cdr.getCallTime())) {
            baseFee = baseFee * 7 / 10; // 夜间 7折
        }
        return baseFee;
    }
    private boolean isNightTime(LocalDateTime time) {
        int hour = time.getHour();
        return hour >= 23 || hour < 7;
    }
}

After the rule change the base rate becomes 12 cents/second, night window shifts to 00:00‑06:00 and discount deepens to 6折.

public class SettlementCalculator {
    public long calculateRoamingFee(CDR cdr, int userType) {
        long baseFee = cdr.getDuration() * 12; // 12分/秒
        if (userType == 2) {
            baseFee = baseFee * 8 / 10;
        }
        // 夜间优惠 00:00-06:00
        if (isNightTime(cdr.getCallTime())) {
            baseFee = baseFee * 6 / 10;
        }
        return baseFee;
    }
    private boolean isNightTime(LocalDateTime time) {
        int hour = time.getHour();
        return hour >= 0 && hour < 6; // 新时段
    }
}

Git diff highlights the numeric and conditional changes.

@@ -1,7 +1,7 @@
 public long calculateRoamingFee(CDR cdr, int userType) {
-    long baseFee = cdr.getDuration() * 10;
+    long baseFee = cdr.getDuration() * 12;
@@ -15,6 +15,6 @@
-    return hour >= 23 || hour < 7;
+    return hour >= 0 && hour < 6;

LLM analysis output (JSON) captures change intent, affected components, detailed test recommendations, and a high risk rating because the modification touches core billing logic.

{
  "change_intent": "将国际漫游基础费率从10分/秒调整为12分/秒，并同步适配VIP折扣、夜间时段折扣规则变更（时段缩窄至00:00-06:00、折扣力度由7折增强至6折），以符合最新资费政策要求。",
  "affected_components": {
    "frontend": [],
    "backend": ["SettlementCalculator.calculateRoamingFee", "国际漫游计费模块（调用方服务）"],
    "database": [],
    "business_rules": ["国际漫游基础费率（10分/秒 → 12分/秒）", "夜间时段定义（23:00-07:00 → 00:00-06:00）", "夜间折扣率（7折 → 6折）", "VIP用户折扣率（保持8折，但基数变更影响最终金额）"]
  },
  "test_recommendations": {
    "regression_tests": ["非VIP用户日间普通通话（验证基础费率12分/秒是否生效）", "VIP用户日间通话（验证12分/秒 × 0.8 = 9.6分/秒）", "非VIP用户原夜间时段23:30-00:30通话（跨旧夜间时段，仅00:00-00:30适用新6折）", "非VIP用户05:00-05:30通话（全量命中新夜间时段，验证6折）", "边界时间00:00:00和06:00:00的CDR处理（毫秒级精度）"],
    "new_tests": ["00:00-06:00外但紧邻的时段测试：23:59-00:01（验证00:00前不打折、00:00起打折）", "跨日夜间通话：23:50-00:10（仅00:00-00:10适用6折）", "VIP + 夜间双重优惠叠加场景（12 × 0.8 × 0.6 = 5.76分/秒）", "时长为0秒、1秒、最大Long值秒的CDR（整数溢出边界）", "CDR时间字段为null或非法格式的异常路径"] ,
    "edge_cases": ["时区未标准化导致夜间判断错误", "系统时钟漂移影响00:00/06:00判定", "高并发下同一秒大量CDR触发竞争", "费率结果出现小数分，需要四舍五入到整数", "VIP标识新增取值（如userType=3）是否兼容"]
  },
  "risk_level": "HIGH",
  "risk_reason": "核心计费逻辑多维度调整直接影响收入准确性；夜间时段收缩与折扣加深易导致漏打折或多打折；结果单位为‘分’，小数处理不当会产生资金差错；无前端或数据库变更，错误直接透传至账单系统，具备强财务敏感性和监管审计风险。"
}

AI vs Manual Analysis Comparison

Time: Manual 30‑60 minutes → AI 5‑10 seconds.

Coverage: Manual relies on personal experience, ~30% omission → AI systematic analysis, <5% omission.

Consistency: Different people produce different results → AI yields standardized output.

Traceability: Hard to record manual reasoning → AI records full prompt and JSON result.

Edge‑case detection: Easy to overlook → AI automatically discovers.