When New LLMs Impress, Their Flaws Quickly Disappoint

Today I evaluated CodeX and GPT5.6‑Sol by creating a directory tree where each sub‑folder contains a TASK.md that instructs the model to rewrite the project as a pure front‑end bundle (only index.html, index.css, index.js) while preserving functionality.

The structure looks like:

├─ agnes-workflow
│   └── TASK.md
├─ llm-eval
│   └── TASK.md
├─ simpledraw
│   └── TASK.md

Running CodeX with this setup produced a screenshot (see below) that listed three tasks, but the model only completed two before stopping.

I suspect the overall task length caused the model to forget the original requirement after finishing the first two subtasks.

Inspecting the intermediate processing revealed that CodeX mistakenly treated the command list as containing only two directories. It also injected the "huashu‑design" skill, which is meant for animation rather than the intended UI‑UX‑PRO skill.

The first command correctly listed three tasks, but the second command only iterated over the first two directories, completely ignoring the third.

Frustrated, I asked CodeX to self‑diagnose the problem; the dialogue is shown in the following image.

Switching to Fable5, I had previously been impressed by its ability to design large frameworks, but when I asked it to make tiny UI adjustments—changing font size, color, or element width—it produced nonsensical results. For a width change, it edited a CSS rule that was later overridden by another stylesheet, so the visual change never took effect.

When I queried the web version of ChatGPT about the same CSS issue, its two answers contradicted each other: one claimed the final width would be 64 px, the other said 22 px should win, exposing inconsistent reasoning.

These observations reminded me of the classic machine‑learning concept of catastrophic forgetting, where a model loses previously learned abilities after training on new tasks. I suspect the failures I saw—missing directory traversal, accidental file deletion, and contradictory language‑model answers—are manifestations of this phenomenon, especially as models become more capable and their hallucinations harder to detect.

In short, the growing power of large language models may be hitting a new bottleneck: the inability to reliably retain basic command‑execution skills and avoid catastrophic forgetting, which could have serious implications if left unchecked.

These notes are personal observations without authority, recorded in the hope of sparking discussion.