Why Amazon Forced Human Approval for AI‑Generated Code—and What It Means for Developers

Background

Guide noticed a claim that Amazon requires senior engineers to approve AI‑generated code before it goes live. After confirming the rumor with official sources, the article examines the incidents that prompted this policy and extracts lessons for developers.

Incident 1 – Kiro "Delete‑and‑Rebuild" Failure

In December of the previous year, Amazon’s AI programming assistant Kiro was tasked with fixing a system vulnerability. The AI decided the best solution was to delete and rebuild the environment, which it executed, affecting only the AWS Cost Explorer service in some China regions.

Amazon later stated the root cause was a user‑permission misconfiguration, not a flaw in the AI itself, emphasizing that the AI was merely a tool given excessive privileges.

Key risk: Permission inheritance allowed the AI to act with the authority of a senior engineer, bypassing the usual dual‑review process.

Incident 2 – 1.6 Million Errors and 120 K Lost Orders

In early March, a deployment error caused Amazon’s North American site to generate 1.6 million error responses, crashing order processing and wiping roughly 120 000 orders. Internal briefings linked the event to AI‑assisted changes, but Amazon clarified the underlying cause was a human configuration mistake that the AI tool accelerated.

Amazon’s Response

Following the two incidents, Amazon launched a 90‑day safety reset plan covering 335 critical systems. The plan introduces three core changes:

Dual‑review: All changes to core systems must be reviewed by two engineers.

Senior engineer approval: Critical changes, especially those assisted by AI, require sign‑off from a senior engineer.

Mandatory tooling: All changes must go through an internal change‑management system (Modeled Change Management) rather than ad‑hoc processes.

Root Cause Analysis

Experts argue the problem is not the AI tool but human practices: code‑review processes were not rigorously enforced, and the speed of AI‑generated changes outpaced manual verification, creating a “scissor‑gap” that led to failures.

Practical Recommendations for Developers

To safely adopt AI‑assisted programming, follow these steps:

1. Enforce Code Review

All AI‑generated code, especially changes affecting core business logic, must undergo manual review.

Implement a dual‑review mechanism for critical systems.

2. Apply the Principle of Least Privilege

Never grant AI direct write access to production environments or high‑risk operations.

Require human confirmation before executing AI‑generated scripts or commands.

Record and approve all changes using the internal change‑management tool.

3. Build a Security Checklist

When reviewing AI‑generated changes, focus on:

Distributed scenarios – data consistency, idempotency, distributed locks, transaction boundaries, compensation logic.

Security vulnerabilities – injection attacks, hard‑coded secrets, missing permission checks.

Performance risks – N+1 queries, missing indexes, connection‑pool leaks, thread explosion.

Observability – adequate logging, monitoring metrics, and traceability.

4. Automate Safeguards

Integrate SAST and SCA tools to scan AI‑generated code for security flaws and dependency risks.

Conduct load testing (e.g., JMeter, Gatling) to verify latency and resource consumption under realistic traffic.

Use chaos engineering to inject failures and test the resilience of AI‑generated components.

5. Constrain AI Output with Specifications

Define team‑wide coding standards and exception‑handling patterns using files such as .cursorrules.

Maintain design contracts (e.g., design.md, spec.md) so AI operates within clear boundaries – a practice known as Spec Coding.

Final Thoughts

AI can dramatically increase developer productivity, but over‑reliance without proper controls leads to higher risk. Balance trust in AI with rigorous governance: enforce code reviews, limit AI privileges, and embed automated safety checks.

By adopting these practices, teams can reap the benefits of AI‑assisted coding while avoiding the pitfalls that caused Amazon’s recent outages.