When Should You Trust AI-Generated Code? A Practical Risk Assessment Guide

Recent community discussions keep Vibe Coding hot, and as a deep user who has shipped several products with it, I found Martin Fowler’s blog post “To vibe or not to vibe” highly relevant, so I’m sharing a translation of its core ideas.

The debate over how much to review AI‑generated code is binary, but the answer is nuanced: it depends on three factors.

1. Probability: How likely is the AI to make a mistake?

Assessing probability involves understanding your tool, the suitability of the use case, and the available context.

Understand your tool

The effectiveness of an AI coding assistant depends on the model, prompt engineering, and integration with your codebase and environment. Closed‑source tools hide many details, so evaluation relies on claimed features and personal experience.

Is the use case suitable for AI?

Consider whether your tech stack is well‑represented in the training data, the complexity of the solution you expect, and the scale of the problem.

Pay attention to context

Context includes the prompts you provide and all information the assistant can access.

Does the AI have sufficient access to your codebase to make informed decisions?

How effective is the tool’s code‑search strategy (full indexing, grep‑style, AST graphs)?

Is your codebase structured and modular, or a tangled “mud ball”?

Does the codebase demonstrate good patterns or contain many hacks?

2. Impact: What are the consequences if the AI errs unnoticed?

Impact depends on the use case—whether you’re doing a spike or production code, on‑call responsibilities, and business criticality.

Self‑check questions include:

Would you deploy this code if you were on call tonight?

Does the code affect many components or consumers?

3. Detectability: Can you notice when the AI makes a mistake?

This concerns the feedback loop: test coverage, strong typing, and visibility of changes.

Familiarity with the codebase and stack increases the chance of spotting anomalies early.

Detectability relies heavily on traditional engineering skills such as test coverage, system knowledge, and code‑review practices, which in turn affect confidence in AI‑driven changes.

Combining the three dimensions: a sliding scale of review intensity

By integrating probability, impact, and detectability, you can calibrate how much supervision is needed. Two extremes illustrate this:

Low probability + low impact + high detectability: Vibe Coding is fine; no review needed if it works.

High probability + high impact + low detectability: Perform intensive review and have fallback measures.

Most situations fall between these extremes.

Traditional skills + new skills

Many assessment questions require traditional engineering expertise, while others demand new AI‑related skills and experience.

Example: Reverse‑engineering a legacy system

In a recent migration project, we used AI to generate detailed specifications of existing functionality.

Probability of error: medium—model often struggles with instructions.

Context limitations: we lacked full code access, especially backend code.

Mitigation: ran prompts multiple times, analyzed decompiled binaries.

Impact: medium—thousands of external partners rely on the system, posing reputation and revenue risk, though the application’s complexity is low.

Detectability: medium—no existing test suite for cross‑validation; we plan to involve domain experts and create functional parity tests.

Without a structured assessment, you risk under‑reviewing or over‑reviewing. Calibrating the approach and planning mitigations helps avoid these pitfalls.

Conclusion

This micro‑level risk assessment will become second nature; the more you use AI, the more intuitive these judgments become, guiding you to trust certain changes while scrutinizing others.

The goal isn’t to slow yourself with checklists but to develop instinctive habits that balance AI’s power with its side‑effects.