Convert Any Text to LLM LoRA in a Single Forward Pass with SHINE

Background

Hypernetwork is a neural network that outputs the parameters of another network. This work trains a hypernetwork that takes arbitrary text as input and directly generates LoRA parameters for a large language model (LLM), enabling conversion with a single forward pass.

Previous hypernetwork approaches were limited to small models and simple architectures, often reusing a small MLP, which restricted expressive power.

By innovating the architecture, the authors create a more expressive hypernetwork that can be scaled with large‑scale training and has practical application potential.

Key Contributions

Practical potential: The method is generic and scalable, providing a new way to inject knowledge into LLMs and adapt them quickly.

Novel architecture: A new hypernetwork design balances parameter size and expressive ability.

Training pipeline: Uses the same pre‑training‑then‑instruction‑fine‑tuning paradigm as LLMs, allowing continuous scaling.

Efficient inference: Only one forward pass is needed; no extra prompt tokens are required.

Continual‑learning perspective: Offers a new direction beyond test‑time training (TTT).

Method Overview

Example

The hypernetwork receives a text, produces LoRA, which when merged with the LLM enables multi‑turn dialogue grounded in that text.

Hypernetwork Architecture

The system consists of two parts: the LLM (shared with inference) and a lightweight M2P Transformer. The text is fed to the LLM with added memory embeddings; hidden states at those positions are collected and concatenated into memory states, which are then processed by the M2P Transformer to output fixed‑size LoRA tensors. A trainable “Meta LoRA” is added to the LLM to improve memory state generation. Only the Meta LoRA, the initial memory embeddings, and the M2P Transformer parameters are learned.

As shown below, the hypernetwork has four stages. Stage 1 runs inside the LLM, stages 2‑4 run inside the M2P Transformer.

The four stages are:

Collect memory states (hidden states at memory‑embedding positions).

Add positional embeddings encoding token position and layer index.

Process memory states with Transformer layers using bidirectional factorization to reduce attention cost.

Reshape the output to form LoRA parameters.

This design aligns semantics to parameters, handles high‑dimensional output, and remains computationally efficient.

Training Procedure and Data

The training follows a “pre‑training – instruction fine‑tuning” pipeline. Pre‑training uses two tasks: reconstruction (generate LoRA from text and recover the original text) and completion (text is truncated and the model must reconstruct and complete it). The authors use 6 B tokens, the largest dataset for hypernetwork‑generated LoRA to date. Instruction fine‑tuning trains the model to answer questions using only the generated LoRA, without feeding the original text.

Experimental Evaluation

Pre‑training Results

Low loss and perplexity on reconstruction indicate that LoRA can almost perfectly memorize the source text; low loss on completion shows some generalization.

Instruction Fine‑tuning

Two stages: multi‑turn QA data then single‑turn QA data. At test time SHINE converts the text to LoRA and answers questions without the text.

Baselines

In‑Context: feed context, prompt, and question.

Naive: only prompt and question.

SFT: generate multiple dialogues per context, temporarily train a same‑size LoRA, then answer.

Gen Adapter: prior work that can generate LoRA from generic text.

Results show SHINE approaches the In‑Context gold standard and outperforms Naive, SFT, and Gen Adapter. Inference time is negligible compared to In‑Context because the text is internalized in parameters.

Comparison with Test‑Time Training (TTT)

SHINE requires only a single forward pass, while TTT needs multiple documents, SFT, RL, and dynamic data generation. SHINE achieves better performance with far lower computational cost.

Scalability

Experiments varying backbone LLM size, LoRA dimension, and M2P Transformer depth all show consistent performance gains, confirming strong scaling properties.

Conclusion and Outlook

SHINE demonstrates that a well‑designed hypernetwork can generate high‑quality LoRA from arbitrary text in one forward pass, enabling efficient knowledge injection and multi‑turn dialogue. The approach scales with data and model size and opens a new avenue for continual learning by turning context into parametric memory. Future work includes handling longer texts, adding reasoning mechanisms, extending to other modalities, and further architectural optimization.