Measuring AI Coding Impact from Individual to Organization with LoongSuite‑Pilot and SLS

The 2026 Google Cloud DORA report highlights that AI‑assisted development can yield a 39% ROI for a 500‑person team, but productivity gains only materialize after a J‑Curve learning period. To determine where an organization stands on this curve, event‑level metrics are essential.

LoongSuite‑Pilot implements the LoongSuite GenAI semantic conventions (an extension of OpenTelemetry GenAI specs) to uniformly collect AI coding agent events—capturing user ID, session ID, agent type, provider, model, token usage, tool calls, and repository information—regardless of the tool (Claude Code, GitHub Copilot, Cursor, etc.). These events are streamed into Alibaba Cloud Log Service (SLS) where a unified fact table stores each agent invocation.

The data model separates the fact table from dimension tables. The dept_user dimension table normalizes organizational hierarchy (department → team) and employee identifiers, allowing flexible joins without embedding organization data in every event. This decoupling enables three engineering benefits: (1) cross‑tool comparability of token counts, (2) session‑level traceability down to individual tool calls, and (3) observability of Skills and Tools via the gen_ai.tool.call.arguments.file_path field.

Using SLS as the analysis layer provides "query‑as‑definition": each chart is backed by a single SQL statement, giving teams immediate control over metrics without product releases. The dashboard is built on a set of common CTEs that pre‑aggregate daily user activity ( active_user) and join with the personnel dimension ( dept_user), ensuring consistent definitions across more than 30 charts.

Section 1 – Core Overview

Top‑level KPI cards show active users, total tokens, session count, and events, each with week‑over‑week comparisons, turning personal adoption into organization‑wide insight.

Section 2 – Structural Distribution

Pie charts break down token share by agent type, model, and provider, answering resource‑allocation questions such as which agent drives most usage and whether tokens are concentrated in a single vendor.

Section 3 – Trends

Time‑series charts display coverage growth, token consumption trends, and per‑agent/token trajectories, revealing whether usage is stabilizing, expanding, or declining.

Section 4 – Department Statistics

Department‑level tables report total tokens, coverage rate, event count, and per‑person token averages, with a LEFT JOIN to surface "registered but not reporting" employees—critical for targeted adoption drives.

Section 5 – Organization & Personnel

Detailed per‑person tables list token totals, event counts, and distinct agents/models used, while agent and model summary tables aggregate usage across the organization.

Section 6 – Skill & Tool

Top‑10 Skill and Tool usage charts are derived from file‑path parsing logic that normalizes various SKILL.md directory structures. The SQL extracts the skill name, counts calls, and aggregates distinct agents, providing a clear view of which practices have truly spread.

Section 7 – Code Repositories

Repository‑level token aggregation identifies the most impacted codebases and distinguishes internal GitLab assets from external GitHub contributions, helping assess alignment with strategic goals.

Section 8 – Token Concentration

Window functions rank users by token consumption, calculating the proportion of tokens held by the top 10 % and visualizing concentration trends over time. This reveals whether high averages are driven by a few power users or broad adoption.

The three‑layer approach—(1) unified semantic collection, (2) flexible SQL‑driven analysis, and (3) actionable organizational insights—turns raw AI‑coding events into measurable business outcomes, allowing R&D leaders to pinpoint adoption gaps, optimize tool portfolios, and drive organization‑wide AI effectiveness.