Run 2.8‑Trillion‑Parameter Models on a 4 GB GPU with AirLLM’s Lossless Inference

GPU‑VRAM bottleneck for ordinary developers

Running state‑of‑the‑art large language models (70 B, 405 B, 671 B, even 2.8 T parameters) normally requires professional GPUs with 80 GB+ VRAM, costing thousands of dollars, or expensive cloud instances. Quantisation to 4‑ or 8‑bit reduces memory but degrades reasoning, long‑text generation and complex question‑answering capabilities.

How AirLLM differs from traditional low‑VRAM solutions

All existing local inference tools (llama.cpp, native Transformers, vLLM, TensorRT‑LLM) load the entire model into GPU memory. When VRAM is insufficient they fall back to either:

CPU‑memory swapping, which causes severe speed loss.

Weight quantisation, which permanently loses model precision.

AirLLM replaces this logic with three self‑developed techniques:

Layer‑wise streaming weight loading : only the currently executed Transformer layer resides in GPU memory; after computation the layer is released and the next layer is streamed from disk. This reduces the memory requirement from the full model size to a single‑layer size.

MoE‑specific expert streaming : for sparse Mixture‑of‑Experts models (e.g., Kimi K3, Mixtral) only the experts activated for the current token (typically 16) are loaded, allowing a 2.8 T‑parameter model to run with just 3.72 GB VRAM.

MXFP4 compressed transfer : weights are 4‑bit compressed for PCIe transfer, then decompressed on‑GPU, cutting disk‑IO traffic by four‑fold while preserving original precision.

Quantitative comparison

Weight precision : traditional 4/8‑bit quantisation incurs permanent loss; AirLLM native mode keeps 100 % original weights (zero loss); optional block‑level compression only compresses data on disk, GPU restores full precision with negligible loss.

VRAM threshold : 70 B model needs ≥16 GB with quantisation, but only 4 GB with AirLLM native mode; block‑level compression still works at 4 GB and adds a ~3× speed boost.

Disk usage : quantisation reduces model file size by ~75 %; AirLLM native mode stores the full original model; block‑level compression halves the file size and speeds up read/write.

Inference latency : quantised models have medium latency; AirLLM native mode shows higher latency due to disk IO (70 B ≈ 2‑3 s/token, Kimi K3 ≈ 292 s/token); block‑level compression reduces latency by 50‑65 % with minor precision loss.

Suitable scenarios : quantised builds target online chat and low‑latency services; AirLLM native mode suits model‑effect verification, academic research and private local testing; block‑level compression balances speed and accuracy for local debugging.

Official hardware‑VRAM matrix (minimum GPU VRAM)

Qwen3 / Mistral (≈8 B): 1‑2 GB

Qwen3‑30B / Mixtral MoE (30‑47 B): 1‑3 GB

Qwen3‑235B MoE (235 B): ≈3 GB

Llama 3.x 70B (native): ≈4 GB

Llama 3.1 405B: ≈8 GB

DeepSeek‑V3 (671 B): ≈12 GB

Kimi K3 MoE (2.8 T): 3.72 GB

Practical installation guide

Prerequisites

CUDA 12 with PyTorch for NVIDIA GPUs (AMD ROCm not supported).

For Kimi K3: install compressed‑tensors, flash‑attn, lock transformers to 4.56.x (CUDA 13 lacks pre‑compiled flash‑attn).

Apple Silicon Macs: install mlx and native Python.

Ensure enough disk space for the split model (same size as the original).

Step 1 – Install core libraries

# Core libraries
pip install airllm transformers==4.56.x torch
# Optional 4/8‑bit speed‑up
pip install -U bitsandbytes
# Extra for Kimi K3 MoE
pip install compressed-tensors flash-attn

Step 2 – Minimal inference script (AutoModel API)

from airllm import AutoModel
MAX_LENGTH = 128
model = AutoModel.from_pretrained(
    "meta-llama/Llama-3-70B-Instruct",
    # compression="4bit",  # enable block‑level compression if desired
    # hf_token="YOUR_HF_TOKEN",  # required for gated Llama models
    prefetching=True,   # overlap IO and compute, ~10% speedup
    delete_original=False  # set True on tight disk space
)
input_text = ["请详细解释大模型逐层流式推理原理"]
input_tokens = model.tokenizer(
    input_text,
    return_tensors="pt",
    truncation=True,
    max_length=MAX_LENGTH,
    padding=False  # avoid padding‑token errors
)
generation_output = model.generate(
    input_tokens["input_ids"].cuda(),
    max_new_tokens=200,
    use_cache=True,
    return_dict_in_generate=True
)
print(model.tokenizer.decode(generation_output.sequences[0]))

Step 3 – Mac Apple Silicon extra dependencies

pip install mlx torch

Common pitfalls and fixes

Safetensor MetadataIncompleteBuffer : caused by insufficient disk space for the split model; free HF cache, enlarge storage, or initialise with delete_original=True to drop the original files.

ValueError: max() arg is an empty sequence : old tutorial called AirLLMLlama2 directly; use AutoModel.from_pretrained for all models.

401 gated model access denied : obtain a HuggingFace token for gated Llama models and pass it via hf_token="xxx".

Tokenizer padding token error : set padding=False as shown in the script.

Operational advice

Do not use AirLLM for real‑time online services; disk‑IO makes 70 B tokens take 2‑3 s and Kimi K3 tokens up to 292 s.

AirLLM does not support fine‑tuning; it lacks gradient back‑propagation.

Prefer NVMe SSDs over SATA or HDDs; the layered loading heavily depends on fast disk reads.

Keep prompts short; longer inputs increase the number of layer loads and exponentially raise latency.

Compatibility limits

Unsupported: AMD ROCm, legacy Tesla K80/M40 GPUs, GGUF model format; only safetensors and native HuggingFace weights are accepted.

Scenario‑based selection

Research / model‑effect verification (4‑GB/8‑GB laptop): use AirLLM native mode, no compression, ample NVMe storage.

Batch document processing with modest speed‑accuracy trade‑off (≥6 GB GPU): enable compression="4bit" for ~3× speed boost and half the disk footprint.

Online high‑throughput AI chat : avoid AirLLM; choose vLLM, TensorRT‑LLM, or llama.cpp quantised builds.

AMD or very old GPUs : fall back to llama.cpp or Ollama.

Strengths and risks

Zero‑loss precision – the only low‑VRAM solution that keeps the original weights intact.

VRAM threshold drops dramatically (4 GB can run 70 B).

Unique MoE optimisation enables 2.8 T‑parameter Kimi K3 on <4 GB VRAM.

Broad model compatibility (Llama, Qwen, DeepSeek, ChatGLM, Baichuan, Mistral, Phi, Gemma).

Simple Transformers‑compatible API (one‑line model switch).

Apache‑2.0 licence – free for commercial use.

Risks: slower inference due to disk IO, single‑maintainer project (maintainer contributes >95 % of commits), incomplete compatibility (AMD, old GPUs, GGUF), high disk usage for lossless mode.

Final verdict

AirLLM fills the gap for developers who need “zero‑cost, lossless testing of gigantic LLMs” on consumer hardware. It is ideal for research, model comparison and privacy‑preserving offline evaluation, but unsuitable for latency‑critical production services where traditional quantised engines remain preferable.

https://github.com/lyogavin/airllm