AI Infra in Practice Part 12: Cross‑Cutting Cost Governance with Token Economics

Problem Statement

A $150,000 monthly AI invoice showed a 40% cost increase but no departmental breakdown. Costs were aggregated by product (OpenAI API $62k, Anthropic API $38k, self‑hosted GPU $41k, vector DB $0.9k). GPU costs are hourly, API costs are per‑token, and there was no unified attribution.

Investigation revealed that 80% of tokens were spent on trivial queries to a flagship model, an embedding endpoint ran idle 24/7 costing $2,500/month, and a RAG pipeline repeatedly re‑embedded the same documents. An Agent workflow triggered 5‑8 LLM calls per user request, inflating the apparent cost of a single request.

This reflects a broader 2026 reality: AI spend is massive, yet most enterprises cannot trace where money is spent or whether it delivers ROI.

Goal

Attribute every AI cost to a specific app , user , and task . Traditional cloud FinOps handles standardized resources (CPU‑hours, GB‑month) with clear ownership; AI workloads span multiple layers (L0‑L8) with heterogeneous billing units, requiring a full‑stack attribution system.

Architecture: Full‑Stack FinOps Cost Attribution

The system consists of five layers (bottom‑to‑top): Cost Collection, Unified Measurement, Three‑Dimensional Attribution, Budget Control, and Cost Reduction.

Cost Collection Layer

Collects raw usage from all layers: GPU metrics via DCGM Exporter & Prometheus (L0), token counts via LLM gateway (L1), embedding dimensions (L2), sandbox launch times (L5), and log storage (L8).

Unified Measurement Layer

Normalizes all units to USD using a dynamic pricing table that syncs vendor rates (e.g., GPT‑5.2 $1.75 per M input tokens, $14 per M output tokens) and estimates self‑hosted model costs (GPU depreciation, electricity, ops).

Three‑Dimensional Attribution Layer

Tags each cost record with app, user, and task identifiers by injecting a Trace Context at request entry. OpenTelemetry GenAI gen_ai.* namespace provides a vendor‑agnostic schema.

Budget Control Layer

Sets token budgets per app/user/task, triggers alerts at 50% usage, auto‑downgrades models at 80%, restricts low‑priority tasks at 95%, and hard‑blocks at 100%.

Cost Reduction Layer

Applies a combination of smart routing, semantic caching, prompt compression, and batch processing, achieving 60‑98% cost reductions when combined.

Key Techniques

Full‑Chain Token Measurement

Every LLM call passes through a gateway that records model name, input/output token counts, timestamps, trace IDs, cache hits, and billing tokens. OpenTelemetry GenAI defines attributes such as gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.usage.output_tokens, ensuring portable metrics.

Output‑to‑input price ratios average 4:1, up to 8:1 for some models, meaning agents that generate long responses incur disproportionate costs. An Agent may trigger 5‑8 LLM calls per task, potentially costing $5‑8 per request.

def meter_llm_call(request, response, trace_context):
    app_id = trace_context.get("app_id")
    user_id = trace_context.get("user_id")
    task_type = classify_task(request.prompt)
    usage = response.usage
    cost = pricing_table.compute(
        model=response.model,
        input_tokens=usage.input_tokens,
        output_tokens=usage.output_tokens,
        cached_tokens=usage.cached_tokens,
    )
    record = CostRecord(
        timestamp=now(), trace_id=trace_context.trace_id,
        app_id=app_id, user_id=user_id, task_type=task_type,
        model=response.model, cost=cost,
        input_tokens=usage.input_tokens, output_tokens=usage.output_tokens,
    )
    cost_store.append(record)
    budget_guard.check(app_id, user_id, cost)
    return record

Three‑Dimensional Cost Attribution

app : unique identifier such as customer-service-bot or code-review-agent.

user : user ID or hashed anonymous ID to spot high‑spending users.

task : categories like simple-qa, code-generation, document-summary, complex-reasoning.

Smart Routing

Routes simple tasks to cheap models (GPT‑4.1 nano, Claude Haiku) saving 95‑99% versus flagship models. A lightweight classifier decides task complexity; routing decisions are themselves measured to avoid hidden costs.

Studies show 60‑80% of queries can be safely routed to small models, cutting 40‑60% of spend with negligible quality impact.

Semantic Cache

Caches responses based on embedding similarity, achieving 2‑10× latency reduction and up to 90% token discount on cache hits. GPTCache (v0.1.44) is the most mature open‑source solution, supporting multiple back‑ends (FAISS, Milvus, Qdrant, Redis Vector).

Risks include cache pollution from adversarial queries; mitigation involves namespace isolation, input sanitization, and strict similarity thresholds.

def cached_llm_call(query, app_id, user_id, threshold=0.95):
    query_embedding = embed_model.encode(query)
    cache_hit = cache_store.search(
        vector=query_embedding,
        namespace=f"{app_id}:{model_version}",
        top_k=1,
        threshold=threshold,
    )
    if cache_hit and cache_hit[0].score >= threshold:
        metrics.increment("cache_hit", app_id, user_id)
        cost_meter.record_cache_savings(app_id, user_id, saved_tokens=cache_hit[0].token_count)
        return cache_hit[0].response
    metrics.increment("cache_miss", app_id, user_id)
    response = llm_gateway.call(query, app_id=app_id)
    cache_store.add(
        vector=query_embedding,
        response=response,
        namespace=f"{app_id}:{model_version}",
        ttl=3600,
    )
    return response

Budget & Quota Enforcement

Implements a four‑tier degradation strategy: 50% usage triggers a warning, 80% auto‑downgrades to a cheaper model, 95% restricts to high‑priority tasks, and 100% blocks all requests.

def enforce_budget(app_id, user_id, task_type, requested_model):
    usage = budget_store.get_monthly_usage(app_id, user_id)
    budget = budget_config.get(app_id, user_id)
    ratio = usage / budget
    if ratio >= 1.0:
        alert.send(f"[BLOCK] {app_id}/{user_id} budget exhausted")
        raise BudgetExhausted("Budget exhausted")
    if ratio >= 0.95:
        if task_type not in HIGH_PRIORITY_TASKS:
            raise BudgetExhausted("Low‑priority task paused")
        return MODEL_DEGRADE_MAP[requested_model]
    if ratio >= 0.80:
        degraded = MODEL_DEGRADE_MAP.get(requested_model, requested_model)
        alert.send(f"[DEGRADE] {app_id} downgraded to {degraded}")
        return degraded
    if ratio >= 0.50:
        alert.send(f"[WARN] {app_id} 50% budget used")
    return requested_model

GPU Utilization Optimization

Detects idle GPUs (utilization <10% for >15 min) via DCGM Exporter and scales clusters elastically based on QPS and latency SLOs. Serverless inference is recommended for low‑frequency services, saving 66‑99% of cost.

Open‑Source Tool Comparison (Summary)

LiteLLM : LLM gateway with automatic cost tracking for 100+ vendors; low‑cost Python proxy; provides key/user/team budgeting and hard‑block.

Langfuse : Full‑chain trace, evaluation, and prompt management; no built‑in budgeting.

OpenTelemetry GenAI : Standardized attribute schema; requires custom measurement logic.

GPTCache : Semantic cache engine; supports multiple vector stores; does not handle cost measurement.

Custom FinOps Dashboard : Fully customizable but higher engineering effort.

Selection guidance: small teams can start with LiteLLM + Langfuse + GPTCache; large enterprises may adopt OpenTelemetry GenAI as the data contract and build a ClickHouse‑backed Grafana dashboard.

Implementation Blueprint

Token Budget Allocation

Top‑down decomposition of an annual AI budget (e.g., $2 M) into quarterly, monthly, weekly, and daily caps, then down‑to‑app, user, and task levels. The first month collects baseline usage; subsequent months set budgets with ±20% tolerance.

GPU Utilization Targets

Aim for ≥70% average GPU utilization; idle detection and auto‑scaling reduce waste (an idle H100 node costs $487/day).

Cost Dashboard

Provides overview, three‑dimensional drill‑down, trend analysis, model distribution, cache benefit, and alert views. Recommended stack: Grafana + ClickHouse.

Metrics & Acceptance Criteria

Cost attribution coverage: 100% (every cost record has app, user, task tags).

GPU utilization: ≥70% (measured via DCGM Exporter → Prometheus).

Semantic cache hit rate: >20% (gateway‑level cache_hit / total_requests).

Prompt cache hit rate: >40% (vendor‑provided cached_tokens / total_tokens).

Budget‑overrun alert coverage: 100% (inject budget checks into each LLM call).

Monthly cost growth: <10% (after accounting for usage growth).

ROI: >3× (business value divided by total AI cost).

Common Pitfalls & Best Practices

Only counting tokens, ignoring GPU costs: Compute full per‑thousand‑token cost for self‑hosted models (depreciation + electricity + ops) and compare with API rates.

Neglecting small‑cost accumulation: Incrementally update embeddings and enforce approval for full re‑embeddings.

Cache slower than direct call: Ensure cache query p95 latency is lower than the cheapest model’s p95 latency; use low‑dimensional embeddings and approximate nearest‑neighbor search.

Routing all tasks to large models: Classify tasks with >95% accuracy before routing 60‑80% of simple queries to small models.

Dashboard not viewed: Push daily cost summaries to Slack, send weekly email reports to owners, and showcase trends in all‑hands meetings.

Cache pollution attacks: Isolate caches by app, user, and model version; sanitize inputs; set strict similarity thresholds.

Overly aggressive degradation: Apply gradual downgrade steps (80% → mid‑tier, 90% → entry‑tier, 95% → limit, 100% → block) with user‑facing notices.

Relationship to Upstream/Downstream Layers

FinOps sits horizontally across L0‑L8, feeding cost data to L9 compliance and providing ROI signals for business decisions.

Summary

Core proposition: attribute every AI cost to app + user + task.

Measurement basis: OpenTelemetry GenAI token standard.

Cost‑cutting trio: smart routing (‑40‑60%), semantic cache (‑20‑40%), prompt cache (‑50‑95%).

Budget control: four‑tier degradation (50% warn → 80% downgrade → 95% limit → 100% block).

GPU governance: idle detection + elastic scaling + serverless fallback (target ≥70% utilization).

Tool stack: LiteLLM gateway + Langfuse observability + GPTCache + Grafana/ClickHouse dashboards.

Key metrics: attribution 100%, GPU ≥ 70%, cache > 20%, ROI > 3×, cost growth < 10%.

Biggest traps: ignoring GPU costs, slow cache, unused dashboards.

Token economics is not merely cost‑saving; it ensures every AI dollar is traceable and delivers measurable returns in a landscape where only 28% of AI projects meet ROI expectations.