Building an Independent QA Audit Loop: A Quality Gate Beyond CI

This lesson adds a second, independent QA audit pipeline on top of the existing CI pipeline.

The CI pipeline already answers whether the code is correct (lint passes, type‑checked, tests green, coverage met). It does not answer three equally important questions: is the code accessible, is it ready for global users, and does it avoid performance‑related failures.

Why QA must be separate from CI

CI validates correctness (code runs, tests pass). QA audit validates reasonableness for all users. The audit runs in parallel with CI, has its own trigger conditions and pass thresholds, and is owned by an independent QA Agent that does not mix into the implementer/reviewer TDD loop.

Three‑dimensional audit model

a11y (accessibility) : WCAG 2.1 AA compliance, scanned with Playwright + axe‑core, expecting zero critical or major violations.

i18n (internationalization) : Detect hard‑coded Chinese strings, dates, and locale identifiers using a custom i18n_audit.py script (regex for CJK characters and common English keys) and the Google i18n‑checker tool, expecting zero un‑translated strings.

Performance security : Core Web Vitals (LCP < 2.5 s, P90 < 5 s) and resource‑usage checks with Lighthouse CI, flagging issues such as N+1 queries, un‑throttled search APIs, or CDN cache drops.

Accessibility audit with Playwright + axe‑core

# Install
pip install playwright
playwright install chromium
npm install @axe-core/playwright  # or pip install axe-core

# Run once against localhost
python -m playwright test --browser=chromium

A shared pytest fixture injects axe after each E2E test, filters known false‑positives, and fails the test suite if any violations remain.

i18n audit script

"""Scan all API responses for hard‑coded user‑visible text"""
import re
from pathlib import Path

HARDCODED_CN = re.compile(r'["\"]([\u4e00-\u9fff]{2,})["\"]')
NON_KEY_EN = re.compile(r'["\"](\b(success|error|failed|invalid)\b)["\"]', re.I)

def scan_file(path: Path) -> list[dict]:
    violations = []
    content = path.read_text(encoding="utf-8")
    for lineno, line in enumerate(content.splitlines(), 1):
        if line.strip().startswith(("#", "'''", '"""')):
            continue
        for m in HARDCODED_CN.finditer(line):
            violations.append({"file": str(path), "line": lineno, "text": m.group(1), "severity": "error", "suggestion": "Wrap in gettext/gettext_lazy call or i18n key"})
        for m in NON_KEY_EN.finditer(line):
            violations.append({"file": str(path), "line": lineno, "text": m.group(1), "severity": "warning", "suggestion": "Use i18n key for user‑facing messages"})
    return violations

def main():
    app_dir = Path("app/")
    all_violations = []
    for py_file in app_dir.rglob("*.py"):
        all_violations.extend(scan_file(py_file))
    if all_violations:
        print(f"
[QA AUDIT] i18n violations: {len(all_violations)}")
        for v in all_violations:
            print(f"  {v['severity'].upper()} {v['file']}:{v['line']} → '{v['text']}'")
        raise SystemExit(1)
    print("[QA AUDIT] i18n: PASS (0 hardcoded strings)")

if __name__ == "__main__":
    main()

Run the script with python scripts/i18n_audit.py. It prints either a pass message or details of each violation.

Performance security audit

Performance security means preventing performance degradation that makes the system unusable, distinct from pure speed optimisation. Example scenarios that belong to performance security include N+1 queries exhausting the DB pool and un‑throttled search APIs being hammered.

Lighthouse CI integration

# .github/workflows/qa-audit.yml (excerpt)
name: QA Audit Pipeline
on:
  pull_request:
    branches: [main]
  workflow_run:
    workflows: ["CI Pipeline"]
    types: [completed]

jobs:
  accessibility-audit:
    runs-on: ubuntu-latest
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install deps
        run: |
          pip install -r requirements.txt
          pip install pytest pytest-cov playwright axe-core-python
          playwright install chromium
      - name: Run a11y audit
        run: pytest tests/e2e/test_auth_a11y.py -v --tb=short

  i18n-audit:
    runs-on: ubuntu-latest
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Run i18n audit
        run: python scripts/i18n_audit.py

  performance-audit:
    runs-on: ubuntu-latest
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    steps:
      - uses: actions/checkout@v4
      - name: Run Lighthouse CI
        uses: treosh/lighthouse-ci-action@v11
        with:
          urls: |
            http://localhost:8000/docs
            http://localhost:8000/api/v1/health
          budgetPath: ./lighthouse-budget.json
          uploadArtifacts: true

The Lighthouse budget ( lighthouse-budget.json) defines performance thresholds such as LCP ≥ 0.8 score and P90 ≤ 2000 ms.

Ledger linkage and sign‑off

# Append to .github/workflows/qa-audit.yml
- name: Update Ledger
  run: |
    FEATURE="$(echo '${{ github.head_ref }}' | cut -d/ -f1)"
    mkdir -p .claude/ledger/$FEATURE
    cat > .claude/ledger/$FEATURE/qa-audit.json << EOF
    {
      "workflow_run_id": ${{ github.run_id }},
      "qa_status": "${{ job.status }}",
      "gates": {"a11y": true, "i18n": true, "performance": true},
      "a11y_violations": 0,
      "i18n_violations": 0,
      "lcp_p90_ms": 1847,
      "completed_at": "${{ github.event.head_commit.timestamp }}"
    }
    EOF
    git add .claude/ledger/$FEATURE/qa-audit.json
    git commit -m "qa: update ledger [skip ci]" || true
    git push || true

Feature merge is allowed only when both ci_status == "pass" and qa_status == "pass".

Self‑check anti‑patterns

A. Running QA audit only once before release leaves the codebase unprotected during development.

B. Embedding i18n audit inside CI without a dedicated pipeline makes it harder to enforce separate quality gates.

C. Treating QA results as mere notifications removes the blocking power needed for reliable releases.

Key insight

The essence of the QA audit loop is to verify that the loop’s output is reasonable for every user. CI checks correctness; QA audit checks reasonableness. Together they form a complete quality loop—without QA, CI can produce a system that discriminates against users.

Next steps

Lesson 0006 will add a security review layer (bandit, semgrep, OWASP ZAP) on top of CI + QA, embed a security gate into the implementer/reviewer TDD loop, and record security findings in the ledger.