Lesson 0005 – QA Audit Loop: Adding an Independent Quality Gate Beyond CI

This lesson focuses on a single task

Extend the CI pipeline from Lesson 0004 into a dual‑track verification system : add an independent QA audit pipeline that checks accessibility (a11y), internationalisation (i18n), and performance—three quality dimensions not covered by CI. The QA audit runs in parallel with CI and together they decide feature sign‑off.

1. Why QA must be independent of CI

Lesson 0004’s CI answers whether the code is correct (lint passes, types are correct, tests are green, coverage meets the threshold). CI does not answer three equally important questions:

Is the code accessible? Can a screen‑reader user complete the registration flow?

Is the code ready for globalisation? Will Chinese usernames, Arabic input, or Emoji themes break?

Is the code performance‑secure? Are there N+1 queries, unbounded rate‑limiting, or memory leaks?

CI 验证正确性 ──→ 代码跑通了吗？测试通过了吗？
    ↓
QA 审计验证合理性 ──→ 行为对所有用户都好吗？

The core principle of the QA audit loop is that it runs independently of CI, has its own trigger conditions and pass thresholds, and is owned by an independent QA Agent rather than the implementer/reviewer TDD loop. QA acts as the "second pair of eyes" for the loop output.

2. Three‑dimensional audit model

QA audit covers three dimensions, each with a specific tool and pass criteria:

a11y (Accessibility) – WCAG 2.1 AA compliance – Playwright + axe‑core – zero critical/major violations.

i18n (Internationalisation) – no hard‑coded strings or dates – i18n‑checker / grep – zero unmarked translation keys.

Performance – Core Web Vitals and resource abuse – Lighthouse CI – LCP < 2.5 s, P90 < 5 s.

3. First track: Accessibility audit

3.1 How Playwright + axe‑core works

axe‑core

(https://github.com/dequelabs/axe-core) is the most widely used open‑source a11y engine. When integrated with Playwright it automatically scans the page for over 150 WCAG rules and outputs violations.

# Install
pip install playwright
playwright install chromium
npm install @axe-core/playwright  # or pip install axe‑core

# Run once manually against localhost
python -m playwright test --browser=chromium

3.2 Embedding a11y checks in E2E tests

Add a shared fixture to the existing E2E test suite:

# tests/e2e/conftest.py
import pytest
from axe_core_python import Axe

@pytest.fixture
def axe(client):
    """Run axe‑core after each E2E test"""
    yield Axe(client.page)
    results = axe.analyze()
    violations = [v for v in results.violations if v["id"] not in ["color-contrast", "label-content-name-mismatch"]]
    if violations:
        pytest.fail(
            f"a11y violations found: {len(violations)} rules violated
" +
            "
".join(
                f"  - [{v['impact']}] {v['description']}" for v in violations
            )
        )

3.3 Adding a11y acceptance criteria to tasks

Append an a11y checklist to every UI‑related task:

## C3 — Implement POST /auth/register

Acceptance criteria:
- [x] AC‑1.1‑1.7 E2E all pass
- [x] axe‑core scan reports zero critical/major violations
- [ ] Screen‑reader test (NVDA + Chrome, Q3 2026)

4. Second track: Internationalisation audit

4.1 Three pitfalls of hard‑coded strings

Identify three kinds of hard‑coded user‑visible text in the task‑management API:

❌ Hard‑coded Chinese: response = {"message": "注册成功"}
❌ Hard‑coded date format: format = "%Y年%m月%d日"
❌ Hard‑coded locale string: locale = "en-US"  # non‑Chinese default

Typical user‑visible strings include API error messages, human‑readable log output, and OpenAPI/Swagger documentation labels.

4.2 i18n audit script

# scripts/i18n_audit.py
"""Scan all API responses for hard‑coded user‑visible strings"""
import re
from pathlib import Path

HARDCODED_CN = re.compile(r'["\']([\u4e00-\u9fff]{2,})["\']')
NON_KEY_EN = re.compile(r'["\'](\b(success|error|failed|invalid)\b)["\']', re.I)

def scan_file(path: Path) -> list[dict]:
    violations = []
    content = path.read_text(encoding="utf-8")
    for lineno, line in enumerate(content.splitlines(), 1):
        if line.strip().startswith(("#", "'''", '"""')):
            continue
        for m in HARDCODED_CN.finditer(line):
            violations.append({"file": str(path), "line": lineno, "text": m.group(1), "severity": "error", "suggestion": "Wrap in gettext/gettext_lazy call or i18n key"})
        for m in NON_KEY_EN.finditer(line):
            violations.append({"file": str(path), "line": lineno, "text": m.group(1), "severity": "warning", "suggestion": "Use i18n key for user‑facing messages"})
    return violations

def main():
    app_dir = Path("app/")
    all_violations = []
    for py_file in app_dir.rglob("*.py"):
        all_violations.extend(scan_file(py_file))
    if all_violations:
        print(f"
[QA AUDIT] i18n violations: {len(all_violations)}")
        for v in all_violations:
            print(f"  {v['severity'].upper()} {v['file']}: {v['line']} → '{v['text']}'")
        raise SystemExit(1)
    else:
        print("[QA AUDIT] i18n: PASS (0 hardcoded strings)")

if __name__ == "__main__":
    main()

Run the script:

python scripts/i18n_audit.py
# Output: [QA AUDIT] i18n violations: 0
# Or:
# ERROR app/routers/auth.py:42 → '注册成功'
# ERROR app/routers/auth.py:78 → '用户名已存在'

4.3 Using Google’s i18n‑checker for stricter audits

npm install -g i18n-checker
i18n-checker ./app --format=json --output=i18n-report.json

5. Third track: Performance‑security audit

5.1 Performance‑security is not performance optimisation

Performance‑security means preventing performance degradation that makes the system unavailable, distinct from chasing raw speed.

Scenario: N+1 queries exhaust DB connection pool – ✅ belongs to performance‑security.

Scenario: Unthrottled search API gets hammered – ✅ belongs to performance‑security.

Scenario: CDN cache hit rate drops from 90 % to 60 % – ✅ belongs to performance‑security.

Scenario: LCP improves from 3 s to 1.5 s – ❌ belongs to performance optimisation.

Scenario: First‑screen image compressed to WebP – ✅ belongs to performance‑security.

The goal of performance‑security is to guard the lower bound : the system must never degrade to an unusable state.

5.2 Integrating Lighthouse CI

Lighthouse CI (https://github.com/GoogleChrome/lighthouse-ci) is the standard way to run Lighthouse in GitHub Actions.

# .github/workflows/qa-audit.yml
name: QA Audit Pipeline
on:
  pull_request:
    branches: [main]
  workflow_run:
    workflows: ["CI Pipeline"]
    types: [completed]

jobs:
  # ─── Gate 4: a11y ───
  accessibility-audit:
    runs-on: ubuntu-latest
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install deps
        run: |
          pip install -r requirements.txt
          pip install pytest pytest-cov playwright axe-core-python
          playwright install chromium
      - name: Start app
        run: uvicorn app.main:app --host 0.0.0.0 --port 8000 &
      - name: Wait for ready
        run: |
          sleep 5 && curl -f http://localhost:8000/health
      - name: Run a11y audit
        run: pytest tests/e2e/test_auth_a11y.py -v --tb=short

  # ─── Gate 5: i18n ───
  i18n-audit:
    runs-on: ubuntu-latest
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Run i18n audit
        run: python scripts/i18n_audit.py

  # ─── Gate 6: Performance ───
  performance-audit:
    runs-on: ubuntu-latest
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    steps:
      - uses: actions/checkout@v4
      - name: Run Lighthouse CI
        uses: treosh/lighthouse-ci-action@v11
        with:
          urls: |
            http://localhost:8000/docs
            http://localhost:8000/api/v1/health
          budgetPath: ./lighthouse-budget.json
          uploadArtifacts: true

5.3 Lighthouse budget configuration

// lighthouse-budget.json
{
  "ci": {
    "collect": {"url": ["http://localhost:8000/docs"]},
    "assert": {
      "categories": {"performance": {"score": [">=0.8"]}},
      "metrics": [
        {"id": "first-contentful-paint", "thresholds": {"pc": 75, "p90": 2000}},
        {"id": "largest-contentful-paint", "thresholds": {"pc": 75, "p90": 2500}},
        {"id": "server-response-time", "thresholds": {"pc": 75, "p90": 800}}
      ]
    }
  }
}

6. QA → Ledger linkage

After the QA pipeline finishes, write the results to a ledger file:

# Append to .github/workflows/qa-audit.yml
- name: Update Ledger
  run: |
    FEATURE="$(echo '${{ github.head_ref }}' | cut -d/ -f1)"
    mkdir -p .claude/ledger/$FEATURE
    cat > .claude/ledger/$FEATURE/qa-audit.json << EOF
    {
      "workflow_run_id": ${{ github.run_id }},
      "qa_status": "${{ job.status }}",
      "gates": {"a11y": true, "i18n": true, "performance": true},
      "a11y_violations": 0,
      "i18n_violations": 0,
      "lcp_p90_ms": 1847,
      "completed_at": "${{ github.event.head_commit.timestamp }}"
    }
    EOF
    git add .claude/ledger/$FEATURE/qa-audit.json
    git commit -m "qa: update ledger [skip ci]" || true
    git push || true

QA sign‑off condition:

ci_status == "pass" AND qa_status == "pass"

7. Self‑check: Identify QA audit anti‑patterns (3 questions)

Determine whether each practice is problematic:

A. "Run the QA audit only once before a release, not during regular development." – __________

B. "Place the i18n audit script directly in CI instead of a separate pipeline." – __________

C. "QA results are merely notifications and do not block merges." – __________

8. Key insight

The essence of the QA audit loop is "verifying that the loop output is reasonable for all users". CI checks correctness; QA audit checks reasonableness. Together they form a complete quality loop. Without QA, CI can produce a system that works but discriminates against certain users.

9. Next step

Lesson 0005’s QA audit covers behavioural correctness, but the task‑management API still has a security risk in JWT authentication and RBAC. Lesson 0006 will address it by:

Adding automated security review (bandit, semgrep, OWASP ZAP) on top of CI + QA.

Embedding a security gate into the implementer/reviewer TDD loop.

Creating a ledger to track security vulnerabilities.

References

axe‑core official repository (2026) – API for Playwright integration.

Lighthouse CI documentation (2026) – budget configuration and GitHub Actions usage.

WebAIM Screen Reader Survey 2025 – automation coverage statistics.

Google i18n‑checker repository (2025) – tool for internationalisation string audits.

Web.dev – Core Web Vitals thresholds (2026).