Running a 2.8‑Trillion‑Parameter K3 Model on 4 GB VRAM with AirLLM

Reevaluating VRAM assumptions

Traditional inference assumes VRAM scales with total parameters; Llama 3.1 405B requires >800 GB in full precision and 60‑80 GB with INT4 quantization.

AirLLM focuses on the amount of parameters resident in VRAM at any moment rather than total model size.

MoE architecture decouples parameter count from memory demand

Mixture‑of‑Experts (MoE) activates only a few experts per token. Kimi K3, with 2.8 trillion parameters, uses only a tiny fraction of weights during a single inference pass.

AirLLM streams weights per‑expert, loading only the experts routed to by the token.

Technical implementation: layer‑wise loading and prefetch parallelism

Layer‑wise weight splitting : On first load, the Hugging Face checkpoint is split into per‑layer files stored in a cache directory.

Dynamic on‑demand loading : After a layer finishes its forward pass, its weights are released; the next layer’s weights are loaded from disk, guaranteeing that at most one layer resides in VRAM.

Prefetch mechanism : Version 2.5 adds asynchronous loading of the next layer while the current layer computes, overlapping I/O with computation and yielding approximately a 10 % speed improvement.

Optional block‑wise quantization : During loading, weights can be compressed to 4‑bit or 8‑bit, reducing data transfer with negligible impact because only stored weights are quantized.

Kimi K3 specific optimizations

pip install compressed-tensors flash-attn

(Flash‑Attention mandatory).

CUDA 12 provides a pre‑compiled wheel for flash‑attn; CUDA 13 requires building from source.

Transformers must be pinned to version 4.56.x because the remote code fails with 5.x.

On a single NVIDIA RTX 6000 Ada (48 GB VRAM), end‑to‑end VRAM usage stabilises at 3.72 GB, lower than Llama 3 8B (~4 GB) despite K3 having a thousand‑times more parameters.

Model memory footprint (no quantization)

Kimi K3 – 2.8 trillion parameters – 3.72 GB VRAM – MoE

DeepSeek‑V3 – 671 billion parameters – ~12 GB VRAM – MoE

Llama 3.1 405B – 405 billion parameters – ~8 GB VRAM – Dense

Qwen3‑235B – 235 billion parameters – ~3 GB VRAM – MoE

Llama 3 70B – 70 billion parameters – ~4 GB VRAM – Dense

Qwen3‑30B – 30 billion parameters – ~1‑3 GB VRAM – MoE

Code example: loading any large model in one line

from airllm import AutoModel

# Initialise model – pass the Hugging Face model ID
model = AutoModel.from_pretrained("Qwen/Qwen3-32B")
# model = AutoModel.from_pretrained("moonshotai/Kimi-K3")
# model = AutoModel.from_pretrained("deepseek-ai/DeepSeek-V3")

input_text = ['What is the capital of United States?']
input_tokens = model.tokenizer(
    input_text,
    return_tensors="pt",
    return_attention_mask=False,
    truncation=True,
    max_length=128,
    padding=False)

generation_output = model.generate(
    input_tokens['input_ids'].cuda(),
    max_new_tokens=20,
    use_cache=True,
    return_dict_in_generate=True)

print(model.tokenizer.decode(generation_output.sequences[0]))

The from_pretrained call supports Llama, Qwen, DeepSeek, Mistral, Phi, Gemma, ChatGLM, Baichuan, InternLM and other major architectures.

Implications for security teams

Local inference removes the need to send sensitive data to external APIs, reducing data‑leakage risk for finance, healthcare, and government sectors.

Conversely, powerful local models expand attackers’ toolkits, enabling AI‑assisted phishing, malware generation, and social‑engineering attacks, requiring updated detection rules and response plans.

Conclusion

AirLLM separates memory demand from model scale through layer‑wise inference and per‑expert streaming, allowing trillion‑parameter models such as Kimi K3 to run in 3.72 GB VRAM without quantization or pruning.

Open‑source repository: https://github.com/lyogavin/airllm

References: AirLLM – GitHub (2026‑07 update); @0x0SojalSec – Kimi K3 VRAM measurements (Twitter/X).