2025 Large Model Engineering Breakthroughs: Cutting Costs, Boosting Performance, and Extending Context

Cost Reduction

DeepSeek‑V3 trained with $5.57 M compute budget (typical >$100 M).

Phi‑4 freezes 99 % of parameters, minimizing computation.

Gemma 3 visual fine‑tuning costs are ten times lower than previous versions.

Performance Improvement

Gemma 3 4B model matches performance of Gemma 2 27B via distillation.

MedGemma gains +20 % on medical benchmarks by training on domain data.

Qwen‑3 adopts a unified “thinking / non‑thinking” architecture.

Memory Efficiency

Gemma 3 reduces KV‑cache memory by 85 % using local/global attention.

Qwen 2.5‑1M supports approximately one‑million‑token context windows.

All models adopt Int4/FP8 quantization for deployment.

Breakthrough Techniques

Auxiliary‑Loss‑Free Load Balancing

In sparse Mixture‑of‑Experts training, a dynamic mechanism adjusts expert bias or routing weights without an auxiliary loss, avoiding gradient interference and improving load balance.

Local / Global Attention

Splits attention into local window attention for short‑range structure and sparse/global attention for long‑range dependencies, combining dense and sparse mechanisms to retain global information with manageable compute.

Mixture‑of‑LoRAs (MoA)

Transforms LoRA adapters into a mixture‑of‑experts structure; multiple LoRA modules are gated per layer or task, enabling multi‑task or multi‑style fine‑tuning while keeping parameter efficiency.

Pan & Scan

Applies a “crop/scale and pan” strategy to input images, preserving native aspect ratios and high resolution; multiple cropped embeddings are mapped to soft tokens, improving OCR of small text and non‑standard aspect ratios.

Multi‑Token Prediction

Generates several future tokens in parallel from the same prefix; consistency losses or gated LoRA ensure coherence, speeding generation and sometimes improving quality in low‑latency settings.

Quantization‑Aware Training (QAT)

Integrates quantization simulation into training or fine‑tuning, allowing models to run at 8/4/2‑bit precision with high accuracy; recent work includes zero‑order QAT, prefixQuant, and scaling‑law‑guided QAT for large models.

Thinking Budget

Introduces an explicit budget signal that predicts remaining “thinking length” and softly guides generation, allocating more inference steps to complex queries while keeping latency low for simple ones; implemented in Google Gemini.

Flow Matching

Treats generation as evolution of a vector field over time; training with flow‑matching objectives enables few‑step or single‑step high‑quality sampling for audio, speech, and audio‑video generation, outperforming traditional diffusion.

Vision‑Language‑Action (VLA) Architecture

Unifies vision, language, and action modules in a single large model, typically built on a pretrained VLM with an added action head or flow‑based decoder, enabling end‑to‑end instruction‑plus‑visual‑observation to action‑sequence mapping.

Partial Improvements

Ring Attention

Original Ring Attention can train sequences >500× longer than prior memory‑efficient methods, exceeding 100 M tokens. The 2025 TokenRing framework adds bidirectional communication and GPU‑network optimizations; RingFormer integrates the mechanism into Conformer to capture local detail and global context.

Post‑Training RLHF – Weight Reward Models

Uses BOND (Best‑N Distillation), WARM (reward‑model ensemble), and WARP (policy ensemble) with RLHF to iteratively optimize preference data. 2025 extensions add synthetic‑data‑driven AI feedback, reducing reliance on human annotation. Gemma 3 adopts WARP as a successor to WARM.

Post‑Training RL – Reward Optimization

GRPO (Group Relative Policy Optimization) computes token‑level loss, contrasting with traditional sequence‑level methods. RLVR employs rule‑based feedback to enhance reasoning. ProRL v2 (NVIDIA) extends RL training for LLMs, achieving state‑of‑the‑art performance on 1.5 B inference models.

Cold Start

RLZero (zero‑shot RL) uses an imagine‑project‑imitate pipeline to turn language or video descriptions into observation sequences, then fits policies, avoiding costly in‑domain RL data. Tsinghua’s Absolute Zero achieves zero‑external‑data self‑evolution with code‑executor validation. Microsoft’s RPT combines multi‑track chain‑of‑thought generation with high‑entropy filtering.

Q‑Filters (Quality Filters)

A context‑agnostic projection evaluates the importance of KV entries and discards low‑importance pairs, compressing KV cache without accessing attention weights; compatible with FlashAttention and reduces memory 2–4×. Also filters zero‑variance prompts and aligns with Int4/FP8 quantization.

Code example

医疗 / 健康 AI (Medical / Health AI)
网络安全 / 负责任 AI (Cybersecurity / Responsible AI)
工程 / 材料 (Engineering / Materials)