Unlocking Compositional Generalization: Meta‑Learning Strategies for Neural Networks

Background

When environments change, biological organisms rely on fast learning mechanisms rather than slow evolutionary processes. Human learners excel because they can recognise shared structure across tasks and recombine a small set of core components to solve many new problems. The paper investigates how these abilities—meta‑learning and compositionality—can be realised in artificial neural networks.

Problem Formulation

Meta‑learning is formalised as a hierarchical optimisation (or sequence‑modelling) problem: an outer optimisation adjusts the learning algorithm itself, while an inner optimisation adapts model parameters on each task. The authors also introduce a formal definition of compositional generalisation , i.e. the capacity to solve novel task combinations after training on a family of tasks that share underlying components.

Algorithmic Contribution

A bilevel‑optimisation meta‑learning algorithm is proposed. Instead of back‑propagating through time or computing costly second‑order derivatives, the method obtains meta‑gradients by:

Training the base learner on a task to obtain parameters θ₁.

Training the same learner on a second, related task to obtain parameters θ₂.

Applying a meta‑plasticity rule that compares the two outcomes (e.g., loss difference or parameter distance) and produces a meta‑gradient that updates the outer‑level optimiser.

This “compare‑and‑update” scheme dramatically simplifies computation while preserving the ability to adapt the learning rule itself.

Architectural Contribution

The study examines modular architectures built with hypernetworks —networks that generate the weights of a target network conditioned on a task embedding. Theoretical analysis shows that, under a set of sufficient conditions (e.g., task embeddings span a linear subspace of the weight space), modular hypernetwork‑based systems can learn strategies that exhibit compositional generalisation. By contrast, monolithic architectures that store a single shared weight matrix for all tasks typically fail to disentangle the reusable components.

Model‑Level Insight

Transformer models are evaluated on sequential compositional tasks. The authors prove a formal equivalence between the Transformer's multi‑head attention mechanism and a hypernetwork: the attention weights act as a latent code that dynamically produces sub‑network parameters. This latent code enables:

Reuse of learned operations across different sub‑tasks.

Recombination of operations to form new task solutions.

Empirical results demonstrate that the structured latent code can accurately predict which sub‑tasks the Transformer will invoke when presented with unseen task combinations, confirming the model’s compositional generalisation capability.

Conclusion

The work deepens the theoretical and empirical understanding of how meta‑learning algorithms, modular hypernetwork architectures, and Transformer‑based latent codes jointly support rapid adaptation and compositional generalisation in neural networks.