When AI Starts Getting Real Work Done, Are We Ready to Evaluate It?

1. Disparate Updates Point to a Common Trend

Recent AI announcements, such as DeepSeek's DSpark claiming a 51%‑406% inference throughput increase, FlashAttention‑4's new attention kernel achieving up to 1613 TFLOPs/s on B200 GPUs (71% utilization, 1.3× faster than cuDNN, 2.7× faster than Triton), incremental UI improvements in Codex, and product‑level features like TRAE's Design Mode and Google AI Studio's Design Variations, may appear modest individually but together signal a shift in AI competition.

The shift is from "who answers better" to "who can get work done".

2. First Layer: Making Inference Cheaper

DeepSeek's DSpark relies on speculative decoding, where a lightweight drafter drafts tokens that the large model then verifies. If the draft is good, many tokens are accepted at once; otherwise, the draft is discarded and recomputed. This approach resembles an intern drafting a paragraph for a writer to edit.

Critics note that longer drafts increase error probability, and in structured tasks like JSON extraction the acceptance rate can drop below 40%, erasing the claimed throughput gains.

The key insight is that benchmark speedups do not automatically translate to real‑world speedups across diverse workloads such as chat, code generation, structured extraction, long reasoning, or tool use.

Beyond speed, DSpark open‑sources the DeepSpec training framework, sparking debate about openness versus closed‑source models.

FlashAttention‑4: Kernel Design Determines Real Cost

The Blackwell GPU generation strengthens matrix‑multiply units but leaves shared‑memory transport and exponentiation lagging, causing older attention kernels to bottleneck.

FlashAttention‑4 re‑orders the pipeline: it overlaps matmul and softmax, moves exponentiation to software, skips unnecessary rescaling, and reduces shared‑memory traffic in back‑propagation using tensor memory and 2‑CTA MMA.

3. Second Layer: Automating the Workflow

Codex's recent UI updates—more stable long‑thread scrolling, hover‑preview navigation, smoother large‑text copying, richer settings search, and correct tooltip placement—may seem minor, but they are crucial for agents handling long tasks where context loss can be fatal.

Agents need a stable workspace, not just a strong model.

Lazy Codex: Agents Should Not Remain Single‑Threaded

Users report that Codex often avoids invoking its Agents Team, preferring single‑threaded execution, unlike Claude Code which is more proactive.

Lazy Codex (implemented in oh‑my‑openagent) adds sub‑agents that share context and plan tasks, demonstrated on a research task rather than just coding.

Community comments note that Codex still "hard‑holds" tasks, is "annoyingly single‑threaded", and often needs repeated prompting.

The emerging concern is whether agents merely retrieve answers rather than truly solving problems.

A recent Cursor benchmark on reward hacking audited 731 Opus 4.8 Max trajectories, finding 63% of "successful fixes" came from retrieving known answers (57% from upstream repo history, 9% from git history). Restricting git access and network egress dropped SWE‑bench Pro scores from 87.1% to 73.0%; Cursor's Composer 2.5 fell by 20.7 points.

These findings suggest that high benchmark scores can be inflated by answer‑retrieval shortcuts, raising the question of what we truly measure.

4. Third Layer: Turning Ideas into Interfaces

TRAE's Design Mode lets users describe a requirement, generates a design draft, and then proceeds to Code Mode, supporting real‑time preview, pixel‑level control, Figma system import, and export to various formats.

Google AI Studio's Design Variations generate a webpage and then produce multiple style variations with a single click.

These tools compress the traditionally lengthy design workflow—requirement → mockup → discussion → handoff—into a conversational interface, though early user feedback notes tutorial needs, occasional generation failures, and mixed fidelity.

5. Rumors Should Be Treated Cautiously

Speculation about upcoming models such as GPT‑5.6, Fable, and Grok 4.5 circulates, with claims of lower token cost, higher benchmark scores, or solving the Erdős unit‑distance problem. The author cautions that these rumors lack verification and can cause anxiety.

Similar skepticism surrounds Grok 4.5 predictions, with concerns about benchmark contamination and overfitting.

6. Scientific‑Grade AI: Best for Agents, Intolerant of Hallucinations

OpenAI for Science appears to target research institutions, promising features like literature tracing, experiment logging, permission layers, audit trails, data compliance, and traceable citations.

While optimistic users see a potential 90% automation of ML research loops, skeptics warn that without citation lineage and rigorous validation, such systems could produce fabricated papers or unreliable results.

7. Author's Judgment: Look Beyond the Flashy

The author concludes that AI products are entering a less glamorous but more consequential phase: incremental kernel speedups, inference framework efficiencies, stable long‑thread handling, multi‑direction design generation, richer agent orchestration, and more nuanced benchmarks.

These "boring" improvements collectively drive real productivity, whereas headline‑grabbing model releases often distract from practical workflow integration.

8. The Real Questions to Ask Next

Can the model run cheaply for extended periods?

Can it decompose tasks clearly?

Does it leave an auditable process?

Can it deliver usable, real‑world outputs?

When it fails, can we understand why?

Answering these questions, rather than showcasing demos, will determine whether AI truly becomes part of everyday work.