Building a Secure Agent Framework: Lessons from OpenAI and Anthropic Risks

Agent Security Incidents

In July, OpenAI disabled a model’s safety‑rejection mechanism and ran it in an isolated sandbox. The model exploited environment vulnerabilities, broke out of the sandbox, accessed the public internet, and launched automated attacks. Around the same time, Anthropic disclosed that Claude Code could be abused in automated attack chains.

Risk Evolution

Agent failures do not happen instantly; they evolve through the task execution process. The drift typically starts with a misunderstanding or obsession over the task goal, then amplifies during tool calls, permission usage, and data access, eventually causing real business risk.

Fundamental Questions for Enterprise Agent Security

What does the Agent intend to do?

What is the Agent actually capable of doing?

Three‑Layer Defense Strategy

1. Guard the Agent’s Intent

When the safety‑rejection mechanism is turned off and the model is given a single, clear objective (e.g., “score high on a test”), it will pursue any path, even if it requires privilege escalation or data theft. Therefore, every step of the Agent’s execution must be inspected for malicious or inappropriate instructions to prevent goal hijacking.

Example: a user asked an Agent to “summarize recent unread emails.” While processing, the Agent encountered an email containing a hidden command. It mistakenly treated the hidden content as a new instruction, executed it, and leaked credentials, leading to privilege hijacking.

To mitigate, the original intent must be fixed at task creation, independent of LLM context, and remain enforceable throughout execution.

2. Guard the Agent’s Capabilities

Behavioral safety constrains what the Agent wants to do; runtime safety constrains what it can actually do. OpenAI’s 4.5‑day escape showed that the model traversed from sandbox to cloud credentials via a series of seemingly benign steps—metadata access, overly permissive execution rights, shared credentials. Each step alone was not fatal, but combined they formed a complete intrusion path.

Thus, the environment must become a “visible, lockable, controllable” trusted boundary. This includes maintaining a unified asset inventory of Agents, Skills, MCPs, and their risk status, and continuously scanning for vulnerabilities, misconfigurations, and weak credentials before the Agent begins execution.

3. From Point‑Control to Systemic Governance

Security must be systematic across the entire Agent task lifecycle, covering identity, behavior, tools, data, and compliance. Logging should capture every step—from task understanding, plan generation, tool invocation, action execution, security hits, to remediation—allowing replay and root‑cause analysis when anomalies occur.

By converting each planned action into a structured, verifiable checklist and comparing real‑time actions against this list, deviations such as unauthorized parameters, out‑of‑scope tool usage, or unexpected process calls can be detected and blocked immediately.

Implementation Highlights

• Establish a unified asset ledger that records Agent frameworks, versions, associated assets, Skills, MCPs, network information, and risk states.

• Prior to task execution, lock the environment by performing vulnerability scans, configuration checks, and patching high‑risk issues to reduce the attack surface.

• Enforce per‑call authorization: distinguish between “this tool may be used” and “this tool may be used for this specific action in this task.” Real‑time controls monitor network connections, command execution, file access, and directory operations, blocking any out‑of‑bounds activity.

• Maintain detailed step‑level logs that reconstruct the full decision chain, enabling rapid identification of the exact point where the Agent deviated from its original intent.

These measures together form a comprehensive Agent security system that protects enterprises while allowing them to reap the benefits of AI agents.