Building a Local LLM with Ollama and a RAG‑Powered Security Knowledge Base

Part 1 – Introduction

The author upgraded to an ASUS ROG 9 notebook equipped with an RTX 5080 GPU, enabling video and image processing without lag and making it feasible to run Ollama locally. After installing Open WebUI for a ChatGPT‑like interface, a local knowledge base was built and the entire setup process is documented.

Part 2 – Technical Research Process

CUDA Overview

CUDA (Compute Unified Device Architecture) is NVIDIA’s GPU computing platform that Ollama uses to accelerate inference on RTX 5080. Ollama automatically detects the GPU, selects the appropriate backend, and handles model loading, GGUF parsing, and VRAM management, shielding users from the “CUDA version hell” of driver‑toolkit‑framework incompatibilities.

Model Quantization

Most models downloaded via Ollama are already quantized for local inference. Quantization reduces numeric precision (e.g., to 4‑bit) to shrink model size and VRAM usage at a modest accuracy cost. Example commands:

ollama pull gemma4:26b

ollama show gemma4:26b

The parameters: 26.5B and quantization: Q4_K_M fields indicate a 26‑billion‑parameter model using the Q4_K_M scheme (≈4‑bit K‑Quant, medium configuration). For a 260‑billion‑parameter model, FP16 weights would require ~52 GB, while 4‑bit quantization drops the theoretical size to ~13 GB, making deployment on consumer GPUs possible.

Installing Ollama

With a proxy (e.g., Proxifier) in place, run one of the following commands to install Ollama:

irm https://ollama.com/install.ps1 | iex

curl -fsSL https://ollama.com/install.sh | sh

After installation, ollama --version confirms the service, which listens on http://localhost:11434 by default.

Downloading Models

Because official mirrors are slow in China, the author recommends using domestic mirrors:

https://github.com/ollamacn

https://mirror.ollama.com

Set the environment variable to point to the mirror:

export OLLAMA_REGISTRY_MIRROR="https://mirror.ollama.com/ollama/"

Example model pulls:

ollama pull gemma4:26b

ollama pull gemma4:31b

For a 16 GB GPU, the 26‑B Q4 version fits comfortably, while the 31‑B version may experience occasional stalls.

Installing Open WebUI

Open WebUI provides a local web UI for managing Ollama models, supporting multi‑model chat, file upload, RAG, web search, and image generation. It can be launched via Docker:

docker run -d -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://192.168.237.111:11434 \
  -v open-webui:/app/backend/data \
  --restart always \
  ghcr.io/open-webui/open-webui:main

When installing via pip, Python 3.11.x is required; other versions raise errors, and the process downloads many dependencies.

Embedding Model for RAG

RAG (Retrieval‑Augmented Generation) combines an embedding model, a vector database, and a LLM. The author uses sentence‑transformers/all‑MiniLM‑L6‑v2 (or nomic‑embed‑text ) to convert documents into semantic vectors (e.g., [0.234, -0.128, 0.556 …]). These vectors are stored in a vector DB; at query time, the user’s question is embedded, matched against stored vectors, and the retrieved passages are fed to the LLM for answer generation.

Creating a Private Security Knowledge Base

In Open WebUI’s “Manage Models” dialog, the author downloads the nomic-embed-text model, selects it as the embedding engine, and enables “Hybrid Search”. A new knowledge base is created via the “Workspace → Knowledge Base → Create” flow, and PDF security documents are imported. The knowledge base is then linked to the gemma4:26b model configuration, with optional features such as internet search, image generation, and code interpreter enabled.

GPU Monitoring

During model loading and inference, nvidia‑smi is used to monitor GPU memory usage, utilization, and running processes, ensuring the model fits into VRAM.

Part 3 – Summary

1. The RTX 5080 laptop runs the Gemma 26B model smoothly; the 31B variant is usable but slower and more VRAM‑hungry. For larger models, a GPU with more VRAM (e.g., RTX 5090) is recommended.

2. Do not expose port 11434 to the public internet; Ollama lacks built‑in authentication, so exposing it can lead to unauthorized GPU usage. If external access is required, place an Nginx reverse proxy with authentication in front.

3. The guide demonstrates a complete offline AI stack: local LLM, Docker‑based Web UI, embedding model, and a RAG‑enabled security knowledge base, enabling private, high‑performance AI queries without relying on cloud services.