When New AI Models Impress, Their Flaws Quickly Disappoint

Today I evaluated CodeX, GPT5.6‑Sol and Fable5 and repeatedly ran into baffling, elementary mistakes that made the impressive capabilities of these new models feel disappointing.

CodeX experiment : I built a directory tree where each sub‑folder contained a TASK.md describing a concrete task (e.g., converting a project to a pure front‑end version). The structure looked like this:

├─ agnes‑workflow
│   └── TASK.md
├─ llm‑eval
│   └── TASK.md
└─ simpledraw
    └── TASK.md

Running the command cat llm‑eval/TASK.md showed three clearly listed tasks, but the execution result displayed only two completed tasks. Screenshots of the task list and the final output confirm the missing third task. The first command processed all three directories correctly; the second command, however, iterated over only the first two, silently ignoring the third, with no obvious reason provided.

I asked CodeX to self‑diagnose the failure, but the subsequent dialogue (omitted here) offered no useful explanation.

Fable5 experiment : While Fable5 performed well on large‑scale design prompts, it struggled with tiny UI adjustments such as changing font size, color, or element width. For a width‑adjustment task, the model generated a CSS rule that was overridden by another stylesheet, leaving the intended change ineffective. The screenshot shows the model suggesting a width of 64 px, yet the final rendered element remained at 22 px. When I queried the web version of ChatGPT about the same issue, its two answers contradicted each other—first claiming 64 px would apply, then stating 22 px should override 64 px.

These contradictory outputs prompted me to recall the classic machine‑learning concept of catastrophic forgetting : after training on a new task, a model can lose previously acquired abilities. I suspect the observed failures—missed directory traversal, accidental file deletions, and basic language inconsistencies—are manifestations of this phenomenon, especially as models become increasingly complex.

Recent literature also notes that as model capability grows, hallucinations become harder for users to detect. Combining undetectable hallucinations with basic command‑execution forgetting may represent a new bottleneck for large language models.

My observations are personal and not authoritative, but I share them in the hope of fostering community discussion about the reliability limits of today’s frontier AI models.