The Second Half of AI: From Model Innovation to Real‑World Utility

Abstract: We are at the midpoint of AI development. Decades of breakthroughs in training methods and models—DeepBlue, AlphaGo, GPT‑4—have shown that search, deep reinforcement learning, and reasoning can solve increasingly complex tasks.

Now, reinforcement learning (RL) finally works in a generalized way: a single approach can handle software engineering, creative writing, IMO‑level math, mouse‑keyboard interaction, and long‑form QA, something that would have been dismissed a year ago.

The "first half" of AI focused on building new models and methods while treating evaluation and benchmarks as secondary. Success was measured by beating world champions in chess, Go, SAT, or winning IOI/IMO medals, but the impact remained limited to academic milestones.

In the "second half," the focus shifts from solving predefined problems to defining useful problems. Evaluation becomes the primary concern: we must ask not only "Can we train a model to solve X?" but also "What should AI be trained to do, and how do we measure real progress?" This requires a product‑manager mindset.

Why the shift? Methods are more exciting and harder than tasks; breakthroughs like AlexNet, Transformers, and GPT‑3 improve many downstream benchmarks because they are general. However, this model‑centric game is reaching its limits.

Solution (the "scheme"): Large‑scale language pre‑training, massive data and compute, and the integration of reasoning and action form a unified approach. From an RL perspective, the three components—algorithm, environment, and prior knowledge—must be balanced. Historically, research emphasized algorithms while treating environment and priors as fixed.

OpenAI’s early work (Gym, World of Bits, Universe) tried to standardize environments, but real progress required strong priors from language models. Pre‑trained models provide world knowledge that, when combined with RL, enable agents to generalize across tasks.

Reasoning (or inference) is a non‑physical action that expands the agent’s decision space. By embedding reasoning as an action, language‑pre‑trained priors let agents select useful strategies even in unseen environments, as illustrated by the ReAct framework.

With the right priors, the specific RL algorithm becomes less critical; the focus moves to designing realistic evaluation settings that reflect real‑world utility.

Challenges for the second half: Existing benchmarks assume i.i.d. tasks and automated evaluation, which diverge from real‑world interactions where agents must cooperate with humans and solve tasks sequentially. New benchmarks should incorporate human‑in‑the‑loop evaluation, long‑term memory, and non‑i.i.d. task distributions.

We must develop utility‑centric evaluation: create tasks that mirror real‑world impact, solve them with the existing scheme, and iterate by enhancing the scheme with new components. This loop will drive research that can build multi‑billion‑dollar products rather than incremental model tweaks.

In summary, the AI "second half" calls for a paradigm shift from model‑centric progress to utility‑centric evaluation, leveraging language‑model priors, reasoning as actions, and realistic environments to achieve truly transformative AI.

Acknowledgements: The blog post is based on the author’s Stanford 224N course and Columbia talks, drafted with assistance from OpenAI Deep Research.