How We Rebuilt Our Code Coverage Platform for Real‑Time, Multi‑Language Insight

1. Introduction: From "Usable" to "Great" – Why We Needed to Revamp the Coverage Platform

The quality team relied on coverage data to gauge test effectiveness, but the existing system suffered from single‑dimensional metrics, coarse testing granularity, and stability issues, prompting a comprehensive overhaul.

2. Multi‑Language Unified “Real‑Time Data Stream” Architecture

Challenge: Unify coverage data generated by heterogeneous languages such as Java and PHP and provide near‑real‑time analysis feedback.

Our Innovative Solution: Design a Kafka‑centric real‑time data stream architecture that eliminates the fragmentation, latency, and format inconsistencies of traditional coverage reports.

Standardized Data Contract

We defined a language‑agnostic coverage data format. Every Agent, whether Java or PHP, must emit records containing branch, commit, traceId, source, method identifier (md5), etc., laying the foundation for unified processing and multi‑dimensional analysis.

{
  "md5": "5e8b9377866dd383dac233a0c9a7",
  "traceId": "api_0156c8522521466c8cf30fad3",
  "branch": "release/r_20240718_v5.7.1",
  "commit": "4f84f27d",
  "className": "BudgetAlarmNoticeJobBean",
  "methodName": "execute",
  "methodDesc": "(Ljava/lang/String;)V",
  "source": "api",
  "data": {
    "28": true,
    "29": true
  }
}

Real‑Time Collection and Push

Agents capture execution data on the fly and push messages to Kafka; server‑side consumers process them instantly, eliminating the long wait for post‑test report generation.

Value: Coverage data becomes a company‑wide, dynamic asset, delivering second‑level feedback after test execution and greatly accelerating issue localization and resolution.

3. From "Line" to "Scenario" – Multi‑Dimensional Fine‑Grained Analysis

Challenge: Traditional line coverage cannot answer precise questions such as which code a specific test case covers.

Our Innovative Solution: Enrich collected data with metadata (especially traceId and source) to achieve unprecedented analysis granularity.

Case‑Level Coverage Tracking: Each execution record carries a traceId; filtering by a specific traceId reconstructs the complete execution path of any single test case.

Precise Attribution of Test Source: The source field distinguishes traffic replay, automated tests, and manual tests, enabling objective evaluation of each testing method.

Method‑Level Precise Statistics: By computing an MD5 hash of the method’s AST, we generate a content‑based unique identifier; any code change updates the MD5, ensuring accurate method‑level coverage.

4. Non‑Intrusive, High‑Stability “Isolated Sandbox” Agent Technology

Challenge: Achieve stable instrumentation and data collection without polluting business applications or causing compatibility issues.

Our Innovative Solution: Develop language‑specific “zero‑impact” agents.

Java Agent – Class Isolation Innovation: Introduce CoverageClassLoader that loads all Agent dependencies (e.g., Kafka client) in an isolated class loader, breaking the parent‑delegation model and guaranteeing absolute runtime isolation from business code.

PHP Agent – “Zero‑Code” Invasion: Leverage the PCOV extension and auto_prepend_file to instrument code without any code changes, drastically lowering integration cost.

5. From "Static Report" to "Real‑Time Trace" – Interactive Visualization

Challenge: Traditional coverage reports are static HTML files generated after tests, with low information density and poor interactivity.

Our Innovative Solution: Combine coverage data with Git information and real‑time execution records to build a dynamic, interactive analysis UI.

Real‑Time Execution Trace Rendering: Using traceId, the frontend highlights executed lines on the code view as the test runs, providing instant visual feedback.

Multi‑Level Filtering & Precise定位: Support filtering by module, package, class, and method, enabling rapid focus on specific code regions.

Multi‑Dimensional Combined Queries & Sub‑Second Reports: Users can combine branch, commit, time range, test source, traceId, etc.; the backend aggregates in real time and produces a precise coverage report within 30 seconds.

6. Platform Impact and Future Outlook

Significant Results:

Scale: Integrated with 590 core services, achieving 99% L1 service coverage.

Efficiency Revolution: Report generation time reduced from 4 minutes per report to 30 seconds for four reports.

Accuracy Leap: Average coverage increased from 74.6% to 86.3%, guiding effective supplemental testing.

Future Plans:

Attribution to Individuals & Teams: Fine‑grained coverage statistics at person/team level.

API‑Level Coverage Association: Deep integration with API management platform for API‑centric views.

Intelligent Test Assistance: Leverage large‑model AI to recommend test cases and proactively warn of weak testing areas.

We believe continuous technical innovation will transform code coverage from a mere metric into the core driver of an intelligent quality assurance era.