Machine Learning Algorithms & Natural Language Processing

Nov 1, 2026 · Artificial Intelligence

LoRA on Decoder Only: Half Parameters Match Full Fine-Tuning on DeepSeek-V4.1-Flash

The article explores optimal LoRA placement for DeepSeek-V4.1-Flash's causal encoder-decoder architecture, finding that applying LoRA only to decoder layers (20–39) with rank 16 matches full-model performance (0.296 vs 0.293 validation loss) while halving adapter parameters and enabling shared KV cache across adapters by freezing layer 20's KV projection.