Ollama vs llama.cpp on Ubuntu: Choose Convenience or Performance? Plus the Context Window Trap

Managing Expectations

Local LLM inference is slower than cloud APIs; its value lies in data privacy, offline use, and zero per‑token cost. Ollama versions update rapidly, indicating high community interest.

Relationship Between Ollama and llama.cpp

llama.cpp is the low‑level inference engine (loads model weights, runs on CPU/GPU, no UI or downloader). Ollama wraps that engine, providing model download, management, and an OpenAI‑compatible API on port 11434. The choice is convenience (Ollama) vs. performance tuning (llama.cpp).

Ollama: Three Commands to Run

Install: curl -fsSL https://ollama.com/install.sh | sh Pull and run a model (e.g., Qwen3 8B): ollama run qwen3:8b Exit with /bye. The local API endpoint:

curl http://localhost:11434/v1/chat/completions 
  -H "Content-Type: application/json" 
  -d '{"model":"qwen3:8b","messages":[{"role":"user","content":"Hello"}]}'

Note: curl | sh trusts the remote script; inspect first or use the official apt repository.

First Pitfall: Default Context Window Is Tiny

Ollama's Modelfile defaults to 2048 tokens; at runtime it adjusts by VRAM:

VRAM < 24 GB → 4096 tokens

24–48 GB → 32768 tokens

> 48 GB → 262144 tokens

Most users land at 4096 tokens, causing long documents to be silently truncated. Check actual context with ollama ps (CONTEXT column).

Three Ways to Increase Context (priority order)

Global default via systemd (recommended): sudo systemctl edit ollama.service Add:

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=32768"

Then:

sudo systemctl daemon-reload && sudo systemctl restart ollama

Per‑model Modelfile :

FROM qwen3:8b
PARAMETER num_ctx 32768

Create and run:

ollama create qwen3-32k -f Modelfile
ollama run qwen3-32k

Per‑request override :

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3:8b",
  "messages": [{"role": "user", "content": "Summarize this doc"}],
  "options": {"num_ctx": 16384}
}'

VRAM cost: KV cache scales linearly with tokens. For an 8B model, ~1 GB per 8K tokens, ~4 GB for 32K, ~16 GB for 128K (on top of model weights). On 8 GB VRAM, 8192 is a safe start; 16 GB can try 16384–32768. Verify with ollama ps after changes.

Rule of thumb: increase context before blaming model quality.

llama.cpp: When You Need Maximum Performance

Ollama's bundled engine lags upstream; a fresh llama.cpp build can be slightly faster — critical for production, benchmarks, or tight hardware.

Compile from source:

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build
cmake --build build --config Release -j

Enable GPU: NVIDIA -DGGML_CUDA=ON, AMD -DGGML_HIP=ON.

Start server:

./build/bin/llama-server -m models/model.gguf -ngl 99 -c 16384 --port 8080

Key flags: -ngl 99 — offload up to 99 layers to GPU (upper bound, not forced full offload); lets you push partial layers on small VRAM. -c 16384 — context length (llama.cpp defaults to only 512, so this is almost always required).

Trade‑off: you manage .gguf model files and versions yourself. Ollama = automatic; llama.cpp = manual.

Performance Determinants: Three Factors

Only three things dictate local inference speed:

VRAM size → maximum model size you can run.

Memory bandwidth → token generation speed.

Model choice → parameter count and quantization.

Hardware tiers (approximate):

CPU only + 16 GB RAM → 7B–8B quantized; slow, barely usable for chat.

8 GB VRAM → 8B–14B quantized; usable for daily tasks.

16 GB VRAM → 14B–32B; comfortable.

24+ GB VRAM → 32B+; near “locally great” threshold.

VRAM decides model size; memory bandwidth decides speed.

Model selection tips:

Parameter count dominates: 7B vs 70B are different leagues.

Quantization: Q4_K_M is the widely accepted balance; Q2 degrades noticeably.

For Chinese tasks, Qwen series consistently outperforms same‑size Llama models.

Download GGUF files from Hugging Face; prefer quantizations by bartowski or mradermacher.

Apple Silicon (unified memory, high bandwidth) is an exception: 16 GB unified memory runs models well.

Author's Recommendation

Start with Ollama — three commands, five minutes, working offline chat.

Immediately raise context to 8192 or higher. Skipping this makes the model seem "dumb" when it's just truncated.

When you hit speed limits or model load failures, switch to llama.cpp for fine‑grained control.

Example split: daily desktop uses Ollama; dedicated inference server runs custom llama.cpp build.

You don't need to pick a side upfront. Get running with Ollama, optimize later.