Run Powerful LLMs Locally on <8GB RAM: Top 10 Small Models & Tools

Most people associate large language models (LLMs) with massive cloud servers and high subscription costs, but recent advances in quantization and model optimization allow these models to run on a laptop or desktop even when RAM or VRAM is under 8 GB.

Decoding Quantization: How Small LLMs Fit Mid‑Range Hardware

The secret is quantization – reducing model weights from 16‑ or 32‑bit floating point to 4‑ or 8‑bit integers, dramatically cutting memory usage without major quality loss. For example, a 7B model that originally needs 14 GB FP16 can run with only 4‑5 GB after 4‑bit quantization.

VRAM vs. RAM: VRAM (GPU) is fast and ideal for LLM inference; RAM is slower but larger. Keep the model in VRAM for best performance.

GGUF format: Preferred quantized model format, compatible with most local inference engines.

Quantization types: Q4_K_M balances quality and efficiency; Q2_K or IQ3_XS save more space but may reduce output quality.

Memory overhead: Allocate 1.2× the model file size to account for activations and prompt context.

Getting Started: Tools for Running Local LLMs

Ollama: A developer‑focused CLI tool for running LLMs locally, scriptable and supports custom models via Modelfile.

LM Studio: A graphical desktop app with chat UI, easy model download from Hugging Face, and simple parameter tweaking.

Llama.cpp: The C++ engine behind many local LLM tools, optimized for GGUF models and supports CPU/GPU acceleration.

Top 10 Small Local LLMs (All Under 8 GB)

1. Llama 3.1 8B (quantized)

Meta’s Llama 3.1 8B is a leading general‑purpose AI model. Quantized versions like Q2_K (3.18 GB, ~7.2 GB RAM) and Q3_K_M (4.02 GB, ~7.98 GB RAM) perform well on chat, code, summarization, and RAG tasks.

ollama run llama3.1:8b

2. Mistral 7B (quantized)

Mistral 7B uses GQA and SWA for top‑tier speed and efficiency. Quantized Q4_K_M (4.37 GB, 6.87 GB RAM) and Q5_K_M (5.13 GB, 7.63 GB RAM) fit comfortably on an 8 GB system, ideal for real‑time chatbots and edge devices.

ollama run mistral:7b

3. Gemma 3:4B (quantized)

Google DeepMind’s Gemma 3:4B runs with Q4_K_M (1.71 GB) using only 4 GB VRAM, suitable for mobile devices and low‑end PCs for text generation, QA, and OCR.

ollama run gemma3:4b

4. Gemma 7B (quantized)

The larger Gemma 7B excels in code, math, and reasoning while still fitting in 8 GB VRAM (Q5_K_M: 6.14 GB, Q6_K: 7.01 GB).

ollama run gemma:7b

5. Phi‑3 Mini (3.8B, quantized)

Microsoft’s Phi‑3 Mini is a compact yet powerful tool for logic, programming, and math. The Q8_0 version (4.06 GB, 7.48 GB RAM) runs fully within an 8 GB limit, perfect for chat and low‑latency tasks.

ollama run phi3

6. DeepSeek R1 7B/8B (quantized)

DeepSeek’s 7B and 8B models are known for inference and coding abilities. R1 7B Q4_K_M (4.22 GB, 6.72 GB RAM) and R1 8B (4.9 GB, 6 GB VRAM) suit 8 GB configurations for enterprise and data‑analysis use cases.

ollama run deepseek-r1:7b

7. Qwen 7B (quantized)

Alibaba’s Qwen 7B is multilingual with a 32K token context. Qwen 1.5 7B Q5_K_M (5.53 GB) and Qwen 2.5 7B (4.7 GB, 6 GB VRAM) are great for chatbots, translation, and coding assistance.

ollama run qwen:7b

8. DeepSeek‑coder‑v2 6.7B (quantized)

Optimized for code generation and understanding, this model runs at 3.8 GB (6 GB VRAM) and is ideal for local code completion tools.

ollama run deepseek-coder-v2:6.7b

9. BitNet b1.58 2B4T

Microsoft’s BitNet uses 1.58‑bit weights, requiring only 0.4 GB memory. Perfect for edge devices, IoT, and CPU‑only inference such as on‑device translation.

ollama run hf.co/microsoft/bitnet-b1.58-2b-4t-gguf

10. Orca‑Mini 7B (quantized)

Built on Llama and Llama 2, Orca‑Mini 7B is versatile for chat, QA, and instruction following. Q4_K_M (4.08 GB, 6.58 GB RAM) and Q5_K_M (4.78 GB, 7.28 GB RAM) are 8 GB‑friendly, ideal for AI agents.

ollama run orca-mini:7b

Conclusion

The models listed above demonstrate that you don’t need a supercomputer to leverage AI; quantization and open‑source innovation enable running advanced LLMs on everyday hardware, offering privacy, lower cost, speed, and flexibility for experimentation and deployment.