How to Build a Lightweight, Scalable AI Code Telemetry System with MCP Architecture

Background

With AI programming assistants rapidly spreading, developers now use many AI tools (CLI, IDE, web) that generate code, but organizations lack a unified way to capture usage, quality, and impact data. Challenges include fragmented data formats, insufficient platform support, and missing metrics for AI code adoption and ROI.

Industry Solutions and Design Thinking

Existing approaches fall into three categories:

Gaode collection : Parses database files for Cursor data; uses MCP diff‑recording for other tools, which introduces noticeable latency and consistency issues.

AIDC VSCode plugin : Requires users to install a heavyweight plugin, limited to Cursor and intrusive.

Our design : Prioritises lightweight, multi‑tool adaptability and standardized output.

Design Principles

Lightweight first : No user‑visible impact; zero‑intrusion.

Diverse adaptation : Tailor collection strategies for CLI, IDE, and web tools.

Standardised output : Convert heterogeneous data into a common schema for downstream analysis.

Reliability : Local caching, retry mechanisms, and version compatibility checks.

Technical Solution

Overall Architecture

The system follows a three‑layer Model Context Protocol (MCP) design:

The layers are:

Collection Layer : Hooks into CLI tools (claude‑code, iflow, codex) and parses IDE snapshots (qoder) to capture raw events.

Processing Layer : Normalises each tool’s data via dedicated Collector components, resolves Git info, and standardises file paths.

Reporting Layer : Batches, retries, and sends normalised events to an SLS log system.

Unified Metrics

Key metric: AI‑Generated Code Ratio (AGCR) = (effective AI‑generated lines / total diff lines) × 100%.

CLI Hook Mechanism

For tools like claude‑code and iflow, a Bash hook script is injected into the user’s configuration. The script validates JSON payloads, trims whitespace, ensures proper JSON structure, generates a UUID, and appends a JSONL record to a date‑based log file.

#!/usr/bin/env bash
set -euo pipefail
CLIENT_TYPE="claude"
# ... (full script omitted for brevity) ...
exec 9>>"$LOG_FILE"
flock 9
printf '%s
' "$RECORD" >&9
exec 9>&-

Telemetry for Gemini/YKCLI/Qwen

Telemetry configuration is added to each tool’s config, directing raw telemetry logs to a local file. The service filters large logs, extracts only the relevant “write‑tool” events, and discards unrelated data to keep I/O efficient.

{
  "telemetry": {
    "enabled": true,
    "target": "local",
    "outfile": "/Users/xxx/.r2c/logs/gemini/raw-telemetry.log",
    "logPrompts": false,
    "useCollector": false
  }
}

Qoder Collector

Qoder stores edit history in History and workspaceStorage directories. The QoderCollector parses these SQLite snapshots, extracts file paths, change types, and diffs, and maintains a SnapshotStore to avoid duplicate reports.

Codex OTLP Collector

Codex supports OpenTelemetry Protocol (OTLP). An HTTP receiver parses incoming OTLP payloads, extracts apply_patch events, and converts them into the unified diff format used by the reporting layer.

Generic Code Report Endpoint

A lightweight /v1/code endpoint accepts JSON payloads (size‑limited, client‑type whitelisted), validates them, normalises paths, classifies the operation (create vs replace), and forwards the result to the standard reporting pipeline.

Service Deployment and Management

The collector runs as a background service. On macOS it registers a LaunchAgent; on Windows it installs a system service via a platform‑service‑manager module. The service creates log directories, sets environment variables, supports log rotation, and can be cleanly uninstalled.

Auto‑Updater

An independent AutoUpdater daemon checks the repository for newer versions hourly. If an update exists, it stops the collector, runs npx @ali/ai-coding-trace@latest to upgrade, validates the new version, and rolls back on failure. The updater starts with a 60‑second delay to avoid lock conflicts.

Platform Compatibility

The system abstracts path handling to work on both macOS and Windows. On Windows logs reside under ProgramData; environment variables ( USERPROFILE, HOME, APPDATA) are auto‑detected. Hook scripts are generated per‑platform, and the service manager adapts installation steps accordingly.

Technical Advantages

Minimal overhead : Hook scripts run in milliseconds; no performance degradation observed.

Zero‑perception : Users need not install plugins or modify workflows.

Robust reliability : Local caching, retries, and native data parsing (e.g., Qoder SQLite) ensure data completeness.

Extensible plugin architecture : Adding a new AI tool only requires implementing a new Collector component.

Summary and Outlook

After several weeks of production use, the system has reliably collected large volumes of AI‑generated code data, enabling analysis of tool adoption, code quality, and workflow optimisation. Future work includes expanding tool coverage, enriching analytics dashboards, and simplifying deployment for broader team adoption.