Essential New Metrics for Monitoring MCP and Tool Calls in API Gateways

Problem Background

Traditional API gateways monitor HTTP status, latency, QPS, error rate, and upstream connections. With large‑scale AI deployments, models now actively invoke external tools, query databases, read/write files, or execute shell commands, turning the simple "input → inference → output" flow into a multi‑hop "input → inference → tool call → inference → output" process.

Applicable Scenarios

AI services that already call external tools (e.g., intelligent customer service, knowledge‑base lookup, code assistants).

Deployments using MCP (Model Context Protocol), OpenAI Function Calling, LangChain/ LlamaIndex agents, or custom tool‑routing layers.

Environments where performance, cost, and security of tool calls must be observable.

Core Knowledge

MCP (Model Context Protocol) standardizes tool invocation: the model returns a JSON payload with tool name and arguments, middleware forwards the request to an MCP server, the server executes the tool, and the result is fed back to the model.

Function Calling (OpenAI), Tool Use (Anthropic), and Agent frameworks (LangChain) follow the same pattern, differing only in protocol format, registration method, and parameter validation.

Traditional API Gateway Monitoring

http_requests_total

– total requests http_request_duration_seconds – request latency http_requests_in_flight – concurrent requests http_response_status – status code distribution http_error_rate – error rate upstream_response_time – upstream latency upstream_status – upstream status code

This model assumes a fixed request‑to‑upstream path and deterministic cost, which no longer holds when a single request may trigger multiple, dynamic tool calls.

Monitoring Blind Spots Introduced by Tool Calls

Call chain length becomes variable; a single request can generate many tool invocations.

Invocation pattern is unpredictable; the model decides which tool, how many times, and in what order.

Tool execution carries its own risks (timeouts, permission errors, parameter validation).

Cost is no longer a simple request count; each tool call incurs token usage and possibly separate billing.

Failure diagnosis is complex because the final error may originate from any intermediate tool.

Identified Gaps in Existing Monitoring

Cannot see which tools were called.

Cannot measure performance distribution of individual tool calls.

Cannot attribute token consumption to specific tools.

Cannot detect abnormal tool‑call patterns (e.g., loops, high frequency).

Cannot audit security risks such as high‑privilege tool usage or sensitive parameter leakage.

Overall Monitoring Approach

The solution introduces a three‑layer observability model, each layer adding specific tags and metrics while sharing a common trace_id to stitch the full call graph.

Layer 1 – User Request

Metric:

ai_gateway_requests_total{method, path, status, has_tool_call}

Latency:

ai_gateway_request_duration_seconds{method, path, has_tool_call, quantile}

Success rate derived from status codes.

Layer 2 – Model Inference

Inference count: ai_model_inference_total{model, trace_id, triggered_tool} Inference latency:

ai_model_inference_duration_seconds{model, triggered_tool, quantile}

Token usage: ai_model_tokens_total{model, trace_id, token_type} Tool‑decision flag:

ai_model_tool_decision_total{model, tool_name, decision}

Layer 3 – Tool Call

Total calls:

ai_tool_call_total{tool_name, tool_category, trace_id, status}

Latency: ai_tool_call_duration_seconds{tool_name, quantile} Parameter size: ai_tool_call_param_size_bytes{tool_name, param_type} Error count: ai_tool_call_error_total{tool_name, error_type} Frequency, retry count, token ratio, and high‑risk flags are also captured.

Security Audit Layer

High‑risk tool calls: ai_tool_call_high_risk_total{tool_name, risk_level} Sensitive parameter detection:

ai_tool_call_param_sensitivity_total{tool_name, contains_sensitive}

Failure‑retry patterns:

ai_tool_call_failure_retry_pattern{tool_name, pattern}

Metric Collection Implementation

1. Gateway Entry Point

Configure Nginx to emit JSON logs with a has_tool_call label and a propagated trace_id. Example log format and Vector configuration are provided to parse the JSON and expose the metrics to Prometheus.

log_format ai_gateway_log escape=json '{"timestamp":"$time_iso8601","trace_id":"$http_x_trace_id","method":"$request_method","path":"$uri","status":$status,"request_time":$request_time,"upstream_response_time":"$upstream_response_time","has_tool_call":"$sent_http_x_has_tool_call"}';

2. Model Middleware

Wrap model API calls in a Python middleware that records start/end timestamps, token usage, and whether a tool call was triggered. The middleware updates Prometheus counters and histograms and logs a structured JSON record containing the trace_id.

import time, logging
from prometheus_client import Counter, Histogram
model_inference_total = Counter('ai_model_inference_total','Total model inference calls',['model','triggered_tool'])
model_inference_duration = Histogram('ai_model_inference_duration_seconds','Model inference duration',['model','triggered_tool'])
model_tokens_total = Counter('ai_model_tokens_total','Total tokens consumed',['model','token_type'])

def call_model_with_metrics(model_name, prompt, trace_id):
    start = time.time()
    response = call_model_api(model_name, prompt)
    duration = time.time() - start
    triggered = 'true' if response.get('tool_calls') else 'false'
    model_inference_total.labels(model=model_name,triggered_tool=triggered).inc()
    model_inference_duration.labels(model=model_name,triggered_tool=triggered).observe(duration)
    model_tokens_total.labels(model=model_name,token_type='input').inc(response['usage']['input_tokens'])
    model_tokens_total.labels(model=model_name,token_type='output').inc(response['usage']['output_tokens'])
    logging.info({"trace_id":trace_id,"model":model_name,"duration":duration,"input_tokens":response['usage']['input_tokens'],"output_tokens":response['usage']['output_tokens'],"triggered_tool":triggered,"tool_calls":response.get('tool_calls',[])})
    return response

3. Tool Routing Layer

Instrument each tool execution in Python. Record call count, latency, input/output payload size, error type, and retry attempts. Ensure the finally block always updates metrics even on failure.

from prometheus_client import Counter, Histogram
tool_call_total = Counter('ai_tool_call_total','Total tool calls',['tool_name','tool_category','status'])
tool_call_duration = Histogram('ai_tool_call_duration_seconds','Tool call duration',['tool_name'])
tool_call_param_size = Histogram('ai_tool_call_param_size_bytes','Tool call parameter size',['tool_name','param_type'])
tool_call_error = Counter('ai_tool_call_error_total','Tool call errors',['tool_name','error_type'])

def execute_tool_with_metrics(tool_name, tool_category, params, trace_id):
    start = time.time()
    status = 'success'
    error_type = None
    try:
        input_size = sys.getsizeof(str(params))
        tool_call_param_size.labels(tool_name=tool_name,param_type='input').observe(input_size)
        result = execute_tool(tool_name, params)
        output_size = sys.getsizeof(str(result))
        tool_call_param_size.labels(tool_name=tool_name,param_type='output').observe(output_size)
        return result
    except TimeoutError:
        status='timeout'; error_type='timeout'; tool_call_error.labels(tool_name=tool_name,error_type='timeout').inc()
        raise
    except PermissionError:
        status='failure'; error_type='permission_denied'; tool_call_error.labels(tool_name=tool_name,error_type='permission_denied').inc()
        raise
    except ValueError:
        status='failure'; error_type='param_error'; tool_call_error.labels(tool_name=tool_name,error_type='param_error').inc()
        raise
    except Exception:
        status='failure'; error_type='execution_error'; tool_call_error.labels(tool_name=tool_name,error_type='execution_error').inc()
        raise
    finally:
        duration = time.time() - start
        tool_call_total.labels(tool_name=tool_name,tool_category=tool_category,status=status).inc()
        tool_call_duration.labels(tool_name=tool_name).observe(duration)
        logging.info({"trace_id":trace_id,"tool_name":tool_name,"tool_category":tool_category,"status":status,"error_type":error_type,"duration":duration,"input_size":input_size,"output_size":output_size})

4. Security Auditing

Maintain a high‑risk tool registry and a list of sensitive keywords. When a tool is invoked, increment ai_tool_call_high_risk_total if it belongs to the high‑risk set, and increment ai_tool_call_param_sensitivity_total when parameters contain any sensitive keyword.

HIGH_RISK_TOOLS = {'execute_shell':'critical','delete_file':'critical','modify_database':'high','send_email':'medium'}
SENSITIVE_KEYWORDS = ['password','passwd','pwd','token','secret','key','api_key','access_key','private_key']

def audit_tool_call(tool_name, params, trace_id):
    if tool_name in HIGH_RISK_TOOLS:
        risk = HIGH_RISK_TOOLS[tool_name]
        tool_call_high_risk.labels(tool_name=tool_name,risk_level=risk).inc()
        logging.warning({"trace_id":trace_id,"audit_type":"high_risk_tool","tool_name":tool_name,"risk_level":risk,"params":params})
    params_str = str(params).lower()
    contains = any(k in params_str for k in SENSITIVE_KEYWORDS)
    tool_call_param_sensitivity.labels(tool_name=tool_name,contains_sensitive='true' if contains else 'false').inc()
    if contains:
        logging.warning({"trace_id":trace_id,"audit_type":"sensitive_param","tool_name":tool_name,"matched_keywords":[k for k in SENSITIVE_KEYWORDS if k in params_str]})

Alert Rules

Prometheus alerting rules cover request success rate, latency, tool‑call ratio, inference loops, token spikes, and security events. Example snippets:

# Request success rate below 95%
alert: AIGatewaySuccessRateLow
expr: (sum(rate(ai_gateway_requests_total{status=~"2.."}[5m])) / sum(rate(ai_gateway_requests_total[5m]))) < 0.95
for: 5m
labels:
  severity: warning
annotations:
  summary: "AI gateway success rate below 95%"
  description: "Current success rate: {{ $value | humanizePercentage }}"

# P95 request latency > 10s
alert: AIGatewayLatencyHigh
expr: histogram_quantile(0.95, rate(ai_gateway_request_duration_seconds_bucket[5m])) > 10
for: 5m
labels:
  severity: warning
annotations:
  summary: "AI gateway P95 latency exceeds 10 seconds"
  description: "Current P95 latency: {{ $value }} seconds"

# Model inference loop detection (more than 10 inferences per trace)
alert: AIModelInferenceLoopDetected
expr: max by (trace_id) (count by (trace_id) (ai_model_inference_total)) > 10
for: 1m
labels:
  severity: critical
annotations:
  summary: "Model inference loop detected"
  description: "trace_id {{ $labels.trace_id }} triggered {{ $value }} inferences"

Implementation Checklist

Instrumentation Points

Gateway: total requests, request latency, status code, has_tool_call flag, trace_id generation.

Model layer: inference count, latency, input/output token usage, tool‑decision flag.

Tool layer: call count, latency, input/output size, error type, retry count, frequency.

Security audit: high‑risk tool calls, sensitive‑parameter detection, failure‑retry patterns.

Alert Priorities

Critical : success rate < 90 %, inference loops, high‑risk tool usage.

Warning : success rate < 95 %, P95 latency > 10 s, tool error rate > 10 %, token consumption spikes, sensitive parameters.

Info : abnormal tool‑call ratio, new tool first‑time usage.

Dashboards

Overall view: request QPS (with/without tool calls), latency percentiles, success rate, top‑10 tools.

Model inference: inference count distribution, token trends, tool‑decision breakdown.

Tool calls: success rate, latency heatmap, error type distribution, data‑size top‑10, high‑risk tool list.

Security audit: high‑risk calls, sensitive‑parameter incidents, retry patterns.

Conclusion and Outlook

The shift to MCP, function calling, and agent architectures requires extending traditional API‑gateway observability from a single‑layer model to a three‑layer framework that captures request‑level, inference‑level, and tool‑level metrics. By propagating a trace_id across all components, operators can reconstruct the full call chain, pinpoint performance bottlenecks, control token‑based costs, and enforce security policies. Future work includes intelligent anomaly detection, cost attribution per user or session, automated performance optimization (e.g., tool call caching), and tighter integration with OpenTelemetry standards.