Deploy Local LLMs Fast with Ollama and AnythingLLM: Step-by-Step Guide
This tutorial walks through installing Ollama, downloading models like Llama 3, configuring AnythingLLM with Ollama as the LLM and embedding provider, setting up LanceDB as the vector database, and creating a workspace for private local AI chat.
Install Ollama
Visit the official Ollama website at https://ollama.com/ and download the installer for your operating system from https://ollama.com/download. Run the installer; on Windows, no virtual machine is required.
Download and Run a Model
After installation, open PowerShell as administrator and type ollama to verify the installation. The command list includes serve, create, show, run, pull, push, list, cp, rm, and help.
Browse the model library at https://ollama.com/library. For example, select llama3. The library page suggests running ollama run llama3. If the model is not present locally, Ollama automatically downloads it. Note: download speeds in China can be slow, sometimes resulting in TLS handshake timeouts; restarting the computer or the Ollama service may help.
After download completes, run ollama list to confirm the model appears. Then start a chat session with ollama run llama3.
Install and Configure AnythingLLM
Download AnythingLLM from https://useanything.com/download (official site: https://useanything.com). Choose the version matching your OS.
Before launching AnythingLLM, start the Ollama service with ollama serve. The default Ollama endpoint is http://127.0.0.1:11434.
Open AnythingLLM and navigate to settings. Configure the following:
LLM Preference : Provider = ollama, URL = http://127.0.0.1:11434, Model = llama3, Token limit = 4096.
Embedding Preference : Provider = ollama, URL = http://127.0.0.1:11434, Model = llama3, Max tokens = 8192 (default).
Vector Database : Keep the default LanceDB.
Create a new workspace (any name). A chat interface will appear on the right, enabling local LLM interaction.
Offline Model Alternative
If model downloads fail, use offline models. On Windows, Ollama stores models at C:\Users\<username>\.ollama\models. A model repository is available at https://modelscope.cn/models.
Personal observation: Llama 3 8B performs poorly in Chinese; Qwen models are recommended for Chinese-language tasks.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
Java Captain
Focused on Java technologies: SSM, the Spring ecosystem, microservices, MySQL, MyCat, clustering, distributed systems, middleware, Linux, networking, multithreading; occasionally covers DevOps tools like Jenkins, Nexus, Docker, ELK; shares practical tech insights and is dedicated to full‑stack Java development.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.