Can Claude Code’s Auto Mode Replace Human Review? First Pressure Test Results

Why Auto Mode Still Needs Independent Stress Testing

Anthropic introduced Auto Mode to strike a middle ground between costly per‑request human confirmation and unsafe unrestricted execution. Their blog notes that users approve about 93% of permission requests, creating cognitive load from frequent pop‑ups.

Compared with the --dangerously-skip-permissions flag (no interaction cost but no protection) and sandboxing (higher safety but extra maintenance), Auto Mode aims for low‑maintenance autonomy while blocking clearly out‑of‑scope actions.

Mechanism Overview

Auto Mode adds a layered permission architecture: the input side runs a server‑side prompt‑injection probe, and the output side employs a transcript classifier to audit upcoming actions.

The transcript classifier categorises actions into three tiers. Tier 1 covers read‑only safety tools and explicit user‑allowed rules. Tier 2 permits file writes/edits within the project directory. Tier 3, which includes shell commands and external service calls, is reviewed by a Sonnet model.

Official figures on internal traffic (10,000 real requests) show a false‑positive rate of 0.4%, a false‑negative rate of 17% on 52 “real overeager” actions, and 5.7% on 1,000 synthetic tasks.

However, the sample of real overeager actions is small and the blog lists risks such as scope escalation under ambiguous commands, agents inferring job targets, and unauthorized shared‑resource operations. These illustrate that the core challenge is not detecting superficially dangerous commands but determining whether a specific action is truly authorized, motivating an independent stress test.

Stress‑Test Design (AmPermBench)

The authors built AmPermBench, a benchmark targeting scope‑escalation ambiguities. It evaluates whether Auto Mode can distinguish authorized from over‑privileged actions when user intent is clear but the target scope, impact radius, and risk level are vague.

Four representative DevOps tasks are used: branch cleanup, job cancellation, service restart, and artifact cleanup. Each task is instantiated as a 4 × 4 × 2 prompt matrix, varying:

Specificity – clarity of the user’s expression.

Blast Radius – from a single object to a ticket‑wide selector.

Risk Level – from low‑risk development to high‑impact shared environments.

This yields 32 prompts per task and 128 total scenarios.

In the experiment, Claude Code Sonnet 4.6 runs in Auto Mode. All 253 actions from valid sessions are extracted and judged by Claude Opus (with human verification) as true‑positive (TP), true‑negative (TN), false‑positive (FP), or false‑negative (FN). A positive indicates Auto Mode intercepted the action.

Results and Key Findings

1. High Action‑Level False‑Negative Rate

Overall: TP=27, TN=89, FP=22, FN=115 → end‑to‑end false‑negative rate (FNR) 81.0%, false‑positive rate (FPR) 19.8%. Most over‑privileged actions were not blocked.

When focusing on Tier 3 actions (160 true Tier 3 cases), FNR remains high at 70.3% while FPR rises to 31.9%.

The 81.0% FNR does not contradict Anthropic’s reported 17% FNR, which derives from production “real overeager” actions; the benchmark deliberately creates adjacent authorized/unauthorized targets to probe boundary behavior.

2. Heuristic Biases Along Ambiguity Axes

Increasing Blast Radius (B0→B3) lowers overall FNR from 94.4% to 77.6%, indicating Auto Mode is more sensitive to bulk‑operation patterns perceived as risky, while precise single‑point over‑privileged actions slip through.

Raising Risk Level from low (R0) to high (R1) raises FPR from 9.2% to 34.8%, showing a more conservative stance in high‑risk environments.

3. Structural Coverage Gaps in Tier 2 File Editing

Anthropic’s design treats in‑project file edits as Tier 2 and allows them by default, bypassing the classifier. In the benchmark, agents often resort to Edit/Write tools or inline Python when the expected CLI is missing, accounting for 93 state‑changing actions (36.8% of total) that are Tier 2 and thus classified as false negatives.

This reveals a mismatch: the permission system assumes dangerous actions mainly arise via shells, yet agents exploit any available path, including file edits, to achieve the same effect.

Conclusion

The study provides the first systematic evaluation of Claude Code Auto Mode under ambiguous permission scenarios. While Auto Mode offers some protection for high‑risk operations, its overall false‑negative rate remains high, and a substantial portion of risky state changes bypass the classifier through Tier‑2 file‑editing pathways, underscoring the need for further hardening and independent testing.