Deploy Open-Source AI Large Models Privately with Ollama – A Step‑by‑Step Guide

Why private deployment

Private deployment keeps source code confidential and enables offline use. Open‑source LLMs (1.5B‑13B parameters) run on modest hardware, while commercial models are more capable but require external services.

Installing Ollama

Ollama is an open‑source tool that bundles pre‑built LLMs and simplifies local deployment. Download the installer from https://ollama.com/ and run the installer; a command prompt opens automatically.

Typical RAM requirements:

7B model – ≥ 8 GB

14B model – ≥ 16 GB

72B model – ≥ 64 GB

Linux installation command: curl -fsSL https://ollama.com/install.sh | sh After installation, ollama help lists available commands.

Ollama commands

ollama list

– show installed models ollama show – display model details ollama pull – download a model ollama push – upload a model ollama ps – view running models ollama cp – copy a model ollama rm – delete a model ollama run <model> – start inference for a model

Pulling and running models

GLM‑4: ollama run glm4 Gemma 2 (9B): ollama run gemma2:9b Qwen 2 (7B): ollama run qwen2:7b After pulling, ollama list confirms the model is installed.

Model storage locations

macOS: ~/.ollama/models Linux: /usr/share/ollama/.ollama/models Windows: C:\Users\<username>\.ollama\models On Linux the default user is ollama; the model directory can be moved by setting OLLAMA_MODELS.

API access

Ollama binds to 127.0.0.1:11434 by default. To allow external access, set the environment variable OLLAMA_HOST=0.0.0.0.

Two compatible endpoints:

Native chat API: http://localhost:11434/api/chat OpenAI‑compatible endpoint: http://127.0.0.1:11434/v1/chat/completions Both can be tested with HTTP clients such as Postman.

AI Hub client

The open‑source AI Hub client aggregates multiple LLM providers (OpenAI, Ollama, Gemini, etc.). Download the release from https://github.com/classfang/AIHub/releases/tag/v1.8.9, launch the Electron‑based app, create a chat window, and select the locally deployed model (e.g., qwen2:7b).

Ollama environment variables

OLLAMA_MODELS – directory for model files (default: user home)

OLLAMA_HOST – listening address (default 127.0.0.1)

OLLAMA_PORT – listening port (default 11434)

OLLAMA_ORIGINS – allowed HTTP origins (comma‑separated, * for unrestricted)

OLLAMA_KEEP_ALIVE – model residency time in memory (e.g., 24h)

OLLAMA_NUM_PARALLEL – concurrent request count (default 1)

OLLAMA_MAX_QUEUE – request queue length (default 512)

OLLAMA_DEBUG – enable debug logs ( 1 for verbose output)

OLLAMA_MAX_LOADED_MODELS – maximum simultaneously loaded models (default 1)

References

1. https://mp.weixin.qq.com/s?__biz=MzUyODk0Njc1NQ==&mid=2247486353&idx=1&sn=b590181589be96cf9e016b39cc4cdbf7

2. https://blog.csdn.net/weixin_40280870/article/details/145275600