How Engram’s Conditional Memory Redefines Sparsity in Large Language Models

Introduction

Engram is an open‑source conditional memory module for large language models (LLMs) that provides a scalable O(1) N‑gram lookup stored in early Transformer layers. By offloading static knowledge to a hash‑based memory, Engram reduces compute in the early layers and frees capacity for reasoning.

Why a New Sparsity Axis?

Mixture‑of‑Experts (MoE) sparsity reduces the number of operations but still forces the model to hard‑memorize entities in the first few layers. Engram introduces a memory‑sparsity axis that replaces hard‑memorization with an extensible hash table, allowing the model to allocate more compute to dynamic inference.

Sparsity Dimension: Parameter sparsity – MoE – activated by top‑k expert routing – suffers from routing overhead and memory fragmentation.

Sparsity Dimension: Memory sparsity – Engram – activated by O(1) hash lookup – requires coupling with computation.

Engram Architecture

Engram is inserted into the middle layers of a Transformer. It retrieves static N‑gram memory and fuses it with the dynamic hidden state.

Three‑Step Pipeline

Tokenizer compression: Apply NFKC normalization and lower‑casing to shrink a 128 k vocabulary to 98 k (23 % reduction).

Multi‑hash head retrieval: Use 2‑gram and 3‑gram tables, each with eight independent hash heads to lower collision probability.

Context gating + lightweight 1‑D convolution: The current hidden state serves as the query, static memory as key/value; the gated output is passed through a SiLU‑activated depthwise convolution.

Multi‑Branch Integration

Engram shares a value matrix with DeepSeek’s multi‑head cache (mHC) across M = 4 branches, while each branch maintains its own key matrix. This design enables fusion of all branches into a single FP8 GEMM on FPGA/TPU.

Sparse Budget Allocation (U‑shaped Scaling Law)

Experiments varying the sparsity budget ρ allocated to MoE show that a pure MoE (ρ = 1) is sub‑optimal. Allocating roughly 20‑25 % of the budget to Engram yields the lowest validation loss.

Budget: 2e20 FLOPs – Optimal ρ: ≈ 0.75 – Pure MoE loss: 1.725 – Mixed loss: 1.710 – Δ = ‑0.015

Budget: 6e20 FLOPs – Optimal ρ: ≈ 0.80 – Pure MoE loss: 1.725 – Mixed loss: 1.711 – Δ = ‑0.014

Scaling to 27 B Parameters

With identical activation compute, Engram‑27B outperforms a MoE‑27B baseline on inference‑heavy tasks more than on pure knowledge tasks, demonstrating a clear architectural advantage.

Long‑Context Experiments (32 k Context Window)

MoE‑27B (50k vocab): Multi‑Query NIAH = 84.2, Variable Tracking = 77.0

Engram‑27B (46k vocab, same loss): Multi‑Query NIAH = 97.0 , Variable Tracking = 87.2

Engram‑27B (50k vocab, same FLOPs): Multi‑Query NIAH = 97.0 , Variable Tracking = 89.0

These results confirm that saved early‑layer compute can be redirected to global attention, improving long‑range reasoning.

Mechanism Analysis

Effective Depth Increase

LogitLens shows faster KL divergence reduction in early layers, and CKA similarity indicates that Engram’s 5th layer behaves like MoE’s 12th layer, effectively deepening the network.

Ablation Study

Removing any of the following components increases loss by more than 1 %: early insertion, context gating, or tokenizer compression.

Functional Split

Zeroing Engram’s output drops TriviaQA accuracy to 29 % while C3 remains at 93 %, suggesting Engram contributes more to knowledge‑heavy tasks than to pure reasoning tasks.

System Efficiency

Placing a 100 B Engram on CPU reduces throughput by only 1.9‑2.8 % on a nano‑vLLM benchmark (512 concurrent requests, 100‑1k token sequences). Deterministic hashing enables prefetch‑friendly access, and Zipf‑distributed keys allow multi‑level caching.

Backbone 4 B: Baseline 9 032 tok/s → +100 B Engram 8 858 tok/s (‑1.9 %)

Backbone 8 B: Baseline 6 316 tok/s → +100 B Engram 6 140 tok/s (‑2.8 %)

Comparison with Related Memory Techniques

N‑gram embedding: OverEncoding, SCONE – only applied at the input layer, no iso‑FLOPs comparison.

Parameter memory: PKM, UltraMem – still rely on network‑based addressing, not O(1) hash.

Non‑parametric retrieval: RETRO, REALM – require external databases and real‑time search at inference.

Conclusion

Engram provides a scalable O(1) conditional memory that reduces early‑layer compute, improves long‑range attention, and yields measurable performance gains across a range of tasks while incurring minimal system overhead.

https://github.com/deepseek-ai/Engram