Heuristic Learning: Reinforcement Without Parameter Updates via .py File

Heuristic Learning (HL) Overview

Heuristic Learning is a reinforcement‑learning paradigm that replaces gradient‑based neural‑network training with iterative editing of a single Python (.py) file that encodes the agent’s decision logic. The editing process is driven by a GPT‑5.4‑powered Codex which evaluates performance, watches failure videos, parses logs, and makes structural code adjustments. Gradient computation is retained only in limited components such as model‑predictive control (MPC) for local search.

Limitations of Traditional Deep Reinforcement Learning

Catastrophic forgetting : new‑task gradients overwrite previously learned weights, preventing continual multi‑task learning.

Black‑box decisions : actions are hidden inside large weight matrices, making reasoning and manual intervention impossible.

Low sample efficiency : massive environment interaction is required for convergence, incurring high computational cost.

Core Idea of HL

HL removes the parameter store entirely. The policy is expressed as readable program code containing explicit state detectors (e.g., “ball is in the upper‑left”), rule logic (e.g., “if ball will land left, move left”), test cases, regression checks, failure logs, and version history. Each iteration Codex evaluates system performance, watches failure videos, parses logs, and makes structural code adjustments. Gradient computation is retained only in limited components such as model‑predictive control (MPC) for local search.

Advantages of the Code‑Centric Approach

Because knowledge is explicit, old abilities are never overwritten; they are packaged as modules and tests that can be invoked, verified, and inherited. This yields inherent interpretability, resistance to forgetting, and higher sample efficiency.

Experimental Results – Atari Games

The HL agent achieved the theoretical maximum of 864 points on Breakout without any neural‑network training. A full Atari‑57 benchmark generated 342 independent coding‑iteration trajectories (two observation modes, three repeats per game). Median performance matched that of PPO, and on several titles such as Breakout, Asterix, and James Bond, HL surpassed human‑player baselines.

Experimental Results – Robot Control

For the quadruped Ant task (8‑joint coordination in a high‑dimensional continuous action space), HL started from basic rhythmic gait rules, then incrementally added posture feedback, ground‑contact sensing, and short‑horizon model‑predictive logic, ultimately scoring over 6000 points—on par with state‑of‑the‑art deep RL models. On the continuous‑control HalfCheetah task, HL attained an average score of 11 836, demonstrating strong adaptability to complex continuous environments.

Recognized Limitations

HL does not yet replace deep networks for raw‑pixel perception tasks such as ImageNet classification. Pure‑Python code without neural networks is currently out of reach for high‑dimensional visual problems.

Future Directions

The next challenge is to fuse neural networks with HL, leveraging HL’s real‑time data‑stream processing to capture reusable online behavior experiences, then converting these explicit rules into high‑quality datasets for periodic neural‑network updates. This hybrid aims to tackle both online learning and continual learning challenges.

References

https://x.com/Trinkle23897/status/2052596837547495549

https://trinkle23897.github.io/learning-beyond-gradients