Unlocking LLM Efficiency: Asymmetry, Token Compression, and Quantization Insights

1. LLM Working Principle

Large language models (LLMs) rely on two fundamental architectures: the Transformer, which uses multi‑layer attention to predict the next token conditioned on previous tokens, and the decoder‑only paradigm, which generates text autoregressively. Both approaches model the probability of the next token based on the context.

2. Long‑Text Token Compression Directions

Improving long‑text capabilities can be achieved with state‑space models such as Mamba combined with external memory modules. These approaches store context externally, activating it only when needed, thereby reducing computation and memory overhead.

First, adopt the Mamba architecture for efficient sequence modeling.

Second, attach an external memory to store and retrieve long‑range context on demand.

3. Asymmetric Phenomena in LLMs

Two key asymmetries are identified:

Token non‑homogeneity : Attention concentrates on the beginning of the sequence and nearby tokens, while middle tokens receive little weight, leading to inefficient long‑context processing.

Keys/Values asymmetry : Adjacent keys are highly correlated (Spearman coefficient ≈ 0.923), enabling their compression with minimal loss, whereas values show lower or even negative correlation, making naïve compression harmful.

4. Parameter Quantization and Scaling Laws

Model parameters are typically stored in FP16, FP32, or BF16 formats. Quantization reduces precision to INT8 or INT4, dramatically shrinking model size. Scaling laws show that performance grows as a power‑law of model size, but hardware limits (e.g., Moore’s law slowdown) motivate quantization to maintain efficiency.

5. Mixed‑Precision Quantization

Mixed‑precision quantization retains a tiny fraction (≈ 0.5 %–1 %) of critical parameters in higher precision (FP16) while converting the majority to low‑precision (INT4). Critical parameters can be identified by weight magnitude, activation statistics (e.g., AWQ), or loss‑impact analysis. This strategy balances memory savings with minimal accuracy loss.

Weight‑based selection: larger weights are kept in high precision.

Activation‑based selection (AWQ): layers with high activation variance retain precision.

Impact‑based selection: parameters whose perturbation most increases loss are preserved.

6. End‑to‑End Mixed‑Precision Quantization Experiments

Experiments on LLaMA‑2 and Vicuna‑bench show that 3/4‑bit quantization can achieve performance comparable to full‑precision models, and in some cases (CherryQ) even surpasses them when evaluated by GPT‑4. A 2‑bit extreme compression maintains reasonable quality but suffers from a limited numeric range.

7. Q&A Summary

The Q&A section addresses practical concerns about quantization training (perceptual quantization vs. post‑training), data volume and distribution effects, mathematical equivalence of Taylor and binomial expansions, and industry‑level challenges such as domain adaptation versus improving general LLM capabilities.