Why Codex Still Breaks Other Code Even With Scope Limits: A Debugging Template
The author compares two Codex implementations of a PyPA build feature, showing how the more concise patch using redirect_stdout inadvertently hijacked another thread's stdout, and provides a three-part template for defining scope, reviewing diffs, and verifying legacy behavior to prevent AI over-engineering.
When asking Codex to implement a small feature, developers often add a constraint like "only change here, keep other functionality unchanged." But how far does that constraint reach? After reading a discussion about Codex over-engineering, the author ran an experiment on a real project: the same small requirement given to Codex twice — first with only the feature description, second with explicit scope and acceptance criteria.
The Small Requirement
The target was PyPA build, a Python packaging tool. It has a --report PATH argument that writes artifact paths, sizes, and SHA-256 hashes to a JSON file. The desired addition: when --report - is passed, write the report directly to stdout so downstream programs can pipe it, e.g.:
python -m build --wheel --report - | jq '.artifacts[].name'In the current version, passing - creates a literal file named "-". Redirecting the report to stdout seems simple, but there are also build logs. A "build succeeded" message goes to stdout, which would break JSON parsing. Both log sources — CLI prompts and the backend subprocess — must be moved to stderr, leaving stdout clean for the report.
Two Runs, Same Tests, Different Diffs
Both runs used Codex CLI 0.169.0 and GPT-6.2 Sol, starting from commit 19b3c67. The first prompt described the feature and existing usage. The second added: only modify the CLI, reuse the existing backend runner and report logic, and run both old and new usage with the same test backend.
CLI Main File ( src/build/__main__.py): First patch +48 lines, -17 lines; Second patch +37 lines, -6 lines.
Files Touched: First patch 3 files; Second patch 4 files (including a design document update).
Repo Tests: Both patches passed 10 tests.
Both patches worked when run standalone: --report - produced valid JSON on stdout, logs went to stderr, and existing --report PATH and --metadata combinations still behaved correctly.
Where the Second Patch Saved Lines
For the backend, both patches used the existing ProjectBuilder.runner parameter to redirect the subprocess stdout to the parent's stderr.
The difference was in handling CLI prompts. The first patch modified the existing _cprint() helper, adding a ContextVar to track whether the report is going to stdout, and conditionally choosing the output stream:
- stream = file or sys.stdout
+ stream = file if file is not None else (sys.stderr if _report_to_stdout.get() else sys.stdout)This works but makes reading the code harder: every _cprint() call requires checking the context variable in main() to know where output goes.
The second patch used Python's built-in contextlib.redirect_stdout():
+ report_stream = sys.stdout if args.report == '-' else None
+ contextlib.redirect_stdout(sys.stderr) if report_stream is not None else contextlib.nullcontext(),This temporarily swaps sys.stdout to stderr during the build, so all scattered print() calls automatically go to stderr without individual changes. Simpler, but sys.stdout is process-wide.
The Hidden Side Effect
Python's documentation warns that redirect_stdout affects other threads using the default print(). Subprocesses are unaffected because they inherit file descriptors at spawn time, so the runner approach for the backend remains necessary.
To expose this, the author wrote a test script that calls build.__main__.main(), pauses the backend after it starts, and has another thread print a marker UNRELATED_THREAD_STDOUT during the build.
Under the first patch, the marker appears on stdout.
Under the second patch, the same marker appears on stderr.
The printing thread never calls the build tool; it merely shares the interpreter's sys.stdout.
The public API is ProjectBuilder; the internal main() was used to make the shared-state effect visible. If always launched as a separate command, both patches pass the earlier checks.
The first patch has its own trade-off: it preserves the other thread's stdout, but that output would mix with the JSON report. The project must decide whether mixed output is acceptable.
Over-Engineering That Doesn't Show in the Diff
Over-engineering often looks like a small request turning into a new framework. GitHub issue https://github.com/openai/codex/issues/56223 describes a cleanup that expanded to eight proposals including migration and compatibility schemes. Community guidelines inspired by Andrej Karpathy suggest checking every change against the original requirement.
In this PyPA case, both changed lines are justifiable and in the expected files. The problem: the affected print() call does not appear in the diff at all. Looking only at "what extra files were changed" misses it.
Karpathy noted that models carry unstated assumptions. Here the assumption: everything written to stdout during the build can be moved together. That holds for standalone commands but breaks when another component shares the process.
This echoes an earlier discussion about AI-generated code: moving a few lines of order-submission code caused a downstream consumer to receive a message before the order existed, a timing bug invisible in the moved code itself.
Some changes have a reason per line, but together they make decisions for other code. Reviewing requires tracing shared state one level deeper to see those callers.
Making "Keep Unchanged" Concrete
Similar to Claude Code's verification method (testing upgrade compatibility with pre-existing data), "other functionality unchanged" needs specifics. The original report file must still work. If the project also requires that other code in the same process keeps writing to stdout, write that down and run it concurrently with the build.
OpenAI's Codex guide recommends specifying goal, context, constraints, and done criteria. Applied here, besides providing the existing report design and runner, the author adds:
Only adjust this build's own prompts and backend logs. In the same process, other code that originally wrote to stdout must still write to stdout.
This yields three concrete checks:
Run --report - standalone; entire stdout parses as JSON.
Run --report PATH; report still writes to file.
Start a build, let another thread print a marker during the build, verify its destination.
If the third check is a real requirement, keep it as a regression test. Following Simon Willison's red/green TDD: first confirm it fails on the second patch, then fix the code, and verify with the same assertion. Next time Codex touches this code, the test will catch regressions.
When using Codex's /review on diffs, include these conditions. Especially for shared-state changes, ask it to trace who else uses that state and run a small scenario. Then the reviewer sees what was checked and what wasn't.
A Reusable Template
Rewriting these checks every time is tedious. The author distilled them into a fill-in template:
[Before Starting: Define Scope]
Let [caller] via [entry] get [new result].
First read [existing implementation, docs, and tests], prefer reusing [existing mechanisms].
This change is limited to [scope].
If existing mechanisms cannot satisfy the need, explain why and what scope must expand.
[Old entry or default usage] must still produce [specific result].
[After Delivery: Inspect Diff]
Against [starting commit or workspace state], check every behavior change serves this need.
Found unrelated refactors, abstractions, or config changes? Explain why they belong in this change.
When touching shared state, public interfaces, or defaults, continue checking who else uses them.
Describe which calls are affected, when they take effect, and where temporary modifications are restored.
[Verify Old Behavior]
Run [new usage], verify [expected result].
Run [old usage] with [original inputs or old data], compare against pre-change results.
If new and old paths share state, add a scenario that uses both simultaneously.
Wait for [signal that new behavior is active], then run [the other path],
verify [output, return value, or file content] matches the original contract.
Deliver diff, actual commands run, and results; note any items not yet verified.For this stdout case, it goes under "who else is using" with the specific other thread's original output destination.
References
Usage notes (Chinese article): https://mp.weixin.qq.com/s/nxTZBVDEB4t2kkfMp9WZsw
PyPA build source, experiment baseline 19b3c67: https://github.com/pypa/build/commit/19b3c671f2ac99815a56e7a1e1307d67fa51d869
PyPA build, report design doc at baseline: https://github.com/pypa/build/blob/19b3c671f2ac99815a56e7a1e1307d67fa51d869/docs/explanation/how-it-works.rst#L267-L281
PyPA build ProjectBuilder API: https://build.pypa.io/en/stable/reference/api.html#build.ProjectBuilder
Python docs, contextlib.redirect_stdout: https://docs.python.org/3.13/library/contextlib.html#contextlib.redirect_stdout
Python docs, subprocess standard streams: https://docs.python.org/3.13/library/subprocess.html#frequently-used-arguments
Andrej Karpathy, notes on coding with Claude: https://x.com/karpathy/status/2015883857489522876
Community collection, andrej-karpathy-skills: https://github.com/multica-ai/andrej-karpathy-skills
Codex GitHub issue #56223: https://github.com/openai/codex/issues/56223
OpenAI, Codex best practices: https://learn.chatgpt.com/guides/best-practices
OpenAI, Codex code review: https://learn.chatgpt.com/docs/code-review#review-local-changes
Simon Willison, Red/green TDD: https://simonwillison.net/guides/agentic-engineering-patterns/red-green-tdd/
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
