Heuristic Learning: A New Reinforcement Learning Paradigm for Continual Learning

Motivation and Observation

Continual Learning remains difficult because neural networks suffer catastrophic forgetting: learning new tasks overwrites old capabilities. The author notes that large‑language‑model (LLM) coding agents can improve software systems without retraining network weights, simply by observing failures, fixing code, adding tests, and replaying executions.

Empirical Findings with Codex

Using Codex (gpt‑5.4) to generate rule‑based agents, the author obtained surprising results on several benchmarks:

Atari Breakout policy scores progressed from 387 → 507 → 839 → 864, reaching the theoretical maximum.

MuJoCo Ant achieved over 6000+ points, comparable to typical Deep RL results.

MuJoCo HalfCheetah reached an average of 11836.7, also matching Deep RL scales.

VizDoom, using only OpenCV/NumPy for screen processing, obtained mean=557.0, min=440.0 across ten seeds.

Across 57 Atari games (342 coding‑agent trajectories), the median Human‑Normalized Score exceeded 1M environment steps, surpassing PPO curves.

These outcomes demonstrate that a coding agent can maintain and grow a software system without neural‑network weight updates.

Defining Heuristic Learning (HL) and Heuristic System (HS)

HL is defined as a learning process whose主体 (主体) consists of program code rather than neural‑network parameters. The key properties are:

HL’s policy is expressed as code, rules, state machines, controllers, or MPC.

It shares the state‑action‑feedback‑update loop with Deep RL, but the update target is the software structure.

Feedback can come from environment rewards, test cases, logs, video replay, or human input.

Updates are performed directly by the coding agent—no back‑propagation.

The maintained object is called a Heuristic System (HS), which includes policy code, state representations, feedback interfaces, experiment records, replay mechanisms, and memory.

Deep RL vs. Heuristic Learning

The author contrasts the two paradigms along several dimensions:

Policy : Deep RL uses neural‑network parameters; HL uses code (rules, state machines, etc.).

State : Deep RL typically uses raw observations; HL uses explicit variables, detectors, or caches.

Action : Deep RL generates actions via a forward pass; HL executes code logic.

Feedback : Deep RL relies on fixed reward signals; HL receives context‑driven feedback such as test failures, logs, or video.

Update : Deep RL updates weights via gradient descent; HL modifies code directly.

Memory : Deep RL may have a replay buffer; HL can store explicit trials, summaries, failure reasons, and version diffs.

Advantages of Heuristic Learning

Explainability – code policies can be read and translated into natural language.

Sample efficiency – a single effective code change can jump to a new policy without gradual learning.

Regression‑testable – old capabilities become test cases, replay logs, or golden traces.

Mitigates catastrophic forgetting – knowledge can be encoded as rules and tests rather than solely in weights.

Why Heuristics Were Historically Neglected

Expert‑system style heuristics were costly to maintain, analogous to pre‑industrial hand‑spinning. Coding agents lower the maintenance curve, turning once‑discarded heuristics into long‑term assets.

Challenges of HL for Continual Learning

HL can still forget:

New rules may fix a failure but break previous scenarios.

New memory entries can steer the agent toward wrong behaviors.

Overly narrow tests cause policies to overfit to test loopholes.

Patches that change public interfaces can silently break callers.

Accumulating rules can become unmanageable for the agent.

To combat these, the author suggests systematic regression testing, golden traces, version diffs, and explicit failure direction documentation.

Coupling Complexity in Heuristic Systems

The author introduces “coupling complexity” as the amount of inter‑related state, rules, tests, feedback, and history an update must handle. It is not measured by lines of code but by how modular, testable, and observable the system is. Stronger models, longer context windows, richer memory, and better tooling increase the feasible coupling complexity.

Towards the Next Paradigm

The author envisions combining HL with neural networks: LLM agents (System 2) generate online experience, HL (System 1) quickly incorporates that experience into code, and periodic retraining updates the neural network with curated data. This hybrid approach aims to solve both online and continual learning problems.

Conclusion

Heuristic Learning reframes the continual‑learning problem from “how to update parameters” to “how to maintain a software system that continuously absorbs feedback.” With coding agents reducing maintenance cost, heuristics that were once impractical become viable components of the next AI paradigm.