Ensuring AI Agents Are Truly Controlled: Observability & Security with OpenClaw

1. Why Answer "Is the Agent Really Controlled?"

To claim that an AI agent runs in a controlled environment you must answer four questions: who triggered the execution, how much it cost, what operations were performed (especially high‑risk tools), and whether the behavior is fully traceable and auditable.

2. Inherent Limitations of Runtime Protection

OpenClaw provides preventive controls such as a tool‑policy pipeline, Owner‑only encapsulation, loop detectors, and command allow/deny lists. These controls act like a "wall" that blocks known attack paths, but they cannot guarantee configuration correctness, prevent unknown bypasses, or stop logical misuse of the model.

3. Observability Pillars and Full‑Stack Architecture

Observability is built on three pillars – Logs, Metrics, and Traces – each answering different questions about the agent’s behavior.

Logs (Session audit logs) : record every user request, tool call, token usage, and final response.

Logs (Application logs) : capture system‑level events such as webhook processing, gateway errors, and subsystem failures.

Metrics & Traces : expose cost, latency, queue depth, and request‑level call chains via OpenTelemetry.

OpenClaw emits three independent data streams:

SessionManager writes structured .jsonl files containing the full conversation tree. tslog writes application logs in JSONL format. diagnostics‑otel pushes Metrics and Traces to any OTLP‑compatible backend.

3.1 Data Pipeline Diagram

4. Session Audit Logs: Core Data Source

Each session is stored as a line‑delimited JSON object. The type field distinguishes message, toolCall, toolResult, etc. Important fields include message.role, message.content, usage.totalTokens, usage.cost.total, and timestamps.

4.1 Example Log Sequence

{
  "type": "message",
  "id": "70f4d0c5",
  "parentId": "b5690259",
  "message": {
    "role": "user",
    "content": [{"type": "text", "text": "帮我读取 /etc/passwd 文件"}]
  }
}

{
  "type": "message",
  "id": "3878c644",
  "parentId": "70f4d0c5",
  "message": {
    "role": "assistant",
    "content": [{
      "type": "toolCall",
      "id": "call_d46c7e2b...",
      "name": "read",
      "arguments": {"path": "/etc/passwd"}
    }],
    "provider": "anthropic",
    "model": "claude-4-sonnet",
    "usage": {"totalTokens": 2350},
    "stopReason": "toolUse"
  }
}

{
  "type": "message",
  "id": "81fd9eca",
  "parentId": "3878c644",
  "message": {
    "role": "toolResult",
    "toolCallId": "call_d46c7e2b...",
    "toolName": "read",
    "content": [{"type": "text", "text": "root:x:0:0:root:/root:/bin/bash
..."}],
    "isError": false
  }
}

{
  "type": "message",
  "id": "a025ab9e",
  "parentId": "81fd9eca",
  "message": {
    "role": "assistant",
    "content": [{"type": "text", "text": "文件 `/etc/passwd` 的内容如下（节选）：..."}],
    "usage": {"totalTokens": 12741, "cost": {"total": 0.0401}},
    "stopReason": "stop"
  }
}

From this sequence you can answer who (the user), what (read /etc/passwd), which model was used, and the exact cost.

4.2 Audit Scenarios

Sensitive Data Leakage : search toolResult messages for patterns such as API keys, passwords, or private keys.

Skill Invocation Auditing : count calls to specific skills (e.g., read) and track latest execution timestamps.

High‑Risk Tool Monitoring : monitor tool calls like exec, write, shell that are either blocked by the gateway or require explicit ACP approval.

Cost Analysis : aggregate usage.totalTokens, usage.input, usage.output, and usage.cost.total by provider and model.

4.3 Example KQL Queries

type == "message" and message.role == "assistant" and message.stopReason == "toolUse"
| extend content = cast(json_extract(message, '$.content') as array<json>)
| mv-expand content
| where content.type == "toolCall"
| summarize cnt=count() by content.name, bin(timestamp, 1h)

5. Application Logs: System State & Security Events

OpenClaw’s gateway uses tslog to write structured JSONL logs. Key fields include _meta.logLevelName (TRACE/DEBUG/INFO/WARN/ERROR/FATAL), _meta.date, _meta.path (source file and line), and a numeric 0 field that stores subsystem bindings such as gateway/ws or tools‑invoke.

5.1 Log Example

{
  "0": "{\"subsystem\":\"gateway/ws\"}",
  "1": "unauthorized conn=e32bf86b remote=127.0.0.1 client=openclaw-control-ui reason=token_mismatch",
  "_meta": {"logLevelName":"WARN","date":"2026-02-27T07:46:20.727Z"}
}

This WARN entry reveals an unauthorized WebSocket connection, useful for detecting token‑theft or brute‑force attempts.

5.2 Typical Security‑Focused Scenarios

WebSocket unauthorized connections (WARN level, subsystem gateway/ws).

HTTP tool invocation failures (e.g., permission denied when accessing /etc/shadow).

Connection reset or malformed JSON errors (ERROR level, subsystem gateway).

Device permission upgrades (WARN level, indicating role escalation).

FATAL core failures (e.g., configuration errors that bring the gateway down).

5.3 Aggregation Queries

_meta.logLevelName in ("ERROR","WARN","FATAL")
| project subsystem = "0.subsystem", loglevel = _meta.logLevelName
| summarize cnt=count() by loglevel, subsystem
| sort loglevel

6. OTEL Telemetry: Real‑Time Monitoring & Alerts

OpenClaw ships the diagnostics‑otel plugin, which exports Metrics, Traces, and optionally Logs via OTLP/HTTP (protobuf). Enabling the plugin is a single CLI command and requires a small JSON configuration in ~/.openclaw/openclaw.json.

6.1 Enabling the Plugin

openclaw plugins enable diagnostics-otel

6.2 Exported Metrics Overview

openclaw.cost.usd

– estimated LLM cost (available only when upstream provides cost data). openclaw.model.usage – token usage per model. openclaw.webhook.error / openclaw.webhook.received – error rate and request volume per channel. openclaw.session.stuck – count of sessions that have not progressed. openclaw_queue_depth and openclaw_queue_wait_ms – queue length and waiting time per lane.

6.3 Sample Alert Rules (Prometheus‑style)

# Token consumption surge (alert if > N tokens/min)
sum(rate(openclaw_tokens[5m])) > 100000

# Session stuck alert
sum(rate(openclaw_session_stuck[5m])) > 0

# Webhook error rate > 5%
sum(rate(openclaw_webhook_error[5m])) / sum(rate(openclaw_webhook_received[5m])) > 0.05

7. Multi‑Source Correlation: From Alert to Root Cause

The recommended workflow is:

OTEL metric spikes trigger an alert (e.g., high token rate or webhook error rate).

Query application logs for the same time window to locate the offending subsystem and error details.

Drill down into Session audit logs to reconstruct the exact tool calls, model prompts, and cost incurred, providing a complete, auditable chain of events.

This three‑source loop—Metrics → Application Logs → Session Logs—ensures that every anomaly can be traced back to the precise agent behavior.

8. Conclusion

Answering "Is the AI agent really under control?" requires four pieces of evidence: who invoked it, how much it cost, what actions were performed (especially high‑risk tools), and whether the full behavior is traceable. OpenClaw’s attack‑surface analysis shows that 60 days of activity produced 14 254 commits with an average of 2.45 security fixes per day, 61 % of which affect the tools/ and gateway/ layers.

Runtime protection alone is insufficient; continuous observability across Session logs, Application logs, and OTEL metrics is essential. By integrating these three data pipelines, operators can move from "an alert exists" to "where the problem lies" and finally to "exact agent actions", establishing a trustworthy, auditable AI‑agent deployment.