How to Build a Production‑Ready AI Chat UI? A Deep Dive into Open WebUI Architecture

Hello, I’m James. In the previous post we discussed how prompt engineering affects AI output quality; this time we look at how a production‑grade AI chat interface is actually built.

If you have run a local model with Ollama but are stuck with a command‑line or a bare‑bones web UI, you may have seen screenshots of a richer interface featuring multi‑model switching, web search, RAG knowledge base, and plugin system—all powered by Open WebUI.

That moment reveals the gap between a "working" UI and a truly production‑ready chat platform, which requires a full architecture.

One‑Sentence Definition

Open WebUI is a full‑stack AI chat platform —the front‑end is built with SvelteKit, the back‑end with Python FastAPI, communication happens via WebSocket and REST API, and it can connect to any model service compatible with the OpenAI API.

Think of it as an AI‑flavored "Notion + Slack" hybrid: Notion‑style document organization (chat history, knowledge base) combined with Slack‑style multi‑user collaboration and permissions, with the model itself acting as the data store.

Why Break Down Its Architecture?

Not every project needs a custom chat UI, but if you want to:

Deploy a private ChatGPT for your team

Embed an AI chat module into an existing product

Understand how streaming rendering, multi‑model routing, and RAG pipelines work in production

The 100 k lines of Open WebUI code serve as a living textbook.

Architecture Layers: Five‑Tier Breakdown

Layer 1: Front‑End Interface

Open WebUI’s front‑end uses SvelteKit, not React—this choice is intentional.

SvelteKit’s compile‑time reactivity outperforms React’s runtime Virtual DOM for streaming rendering: tokens arrive one by one, the UI updates frame‑by‑frame without the overhead of reconciliation.

<!-- Streaming message rendering core logic -->
<script lang="ts">
  import { onMount } from 'svelte';
  export let messageId: string;
  let content = '';
  let isStreaming = false;
  // EventSource receives SSE stream
  onMount(() => {
    const eventSource = new EventSource(`/api/stream/${messageId}`);
    eventSource.onmessage = (event) => {
      const data = JSON.parse(event.data);
      if (data.done) {
        isStreaming = false;
        eventSource.close();
        return;
      }
      // Directly append to string, Svelte auto‑triggers DOM update
      content += data.delta;
      isStreaming = true;
    };
    return () => eventSource.close();
  });
</script>

<!-- Svelte reactive update: re‑render when content changes -->
<div class="message-content">
  {#if isStreaming}
    <StreamingIndicator />
  {/if}
  <MarkdownRenderer {content} />
</div>

Comparison with a React implementation shows that React’s per‑token re‑render can cause noticeable UI jitter when token rates reach 50‑100 tokens/s.

// ❌ React version problem: each token triggers a re‑render
// When token speed is high, UI may jitter
const [content, setContent] = useState('');

useEffect(() => {
  const eventSource = new EventSource(`/api/stream/${messageId}`);
  eventSource.onmessage = (e) => {
    const data = JSON.parse(e.data);
    setContent(prev => prev + data.delta); // each setState causes re‑render
  };
}, [messageId]);

// ✅ Using useRef + manual DOM update to avoid re‑render
const contentRef = useRef('');
const domRef = useRef<HTMLDivElement>(null);

eventSource.onmessage = (e) => {
  const data = JSON.parse(e.data);
  contentRef.current += data.delta;
  if (domRef.current) {
    domRef.current.innerHTML = marked(contentRef.current); // bypass React
  }
};

Key point: streaming rendering should bypass the framework’s reactive system—at 50‑100 tokens/s the framework overhead can cause UI stalls.

Layer 2: API Gateway

The back‑end does not connect directly to the model; a Pipe layer handles routing and interception.

# backend/open_webui/routers/openai.py
# Simplified core routing logic
from fastapi import APIRouter, Request, Depends
from fastapi.responses import StreamingResponse

router = APIRouter()

@router.post("/chat/completions")
async def chat_completion(request: Request, body: ChatCompletionRequest, user = Depends(get_current_user)):
    # 1. Permission check
    if not await check_model_access(user, body.model):
        raise HTTPException(403, "No access to this model")
    # 2. Route request through Pipe layer
    # Pipe decides whether to forward to Ollama, OpenAI, or a custom endpoint
    model_handler = await get_model_handler(body.model)
    # 3. Stream response passthrough
    async def generate():
        async for chunk in model_handler.stream(body):
            # Convert each chunk to SSE format
            yield f"data: {chunk.json()}

"
        yield "data: [DONE]

"
    return StreamingResponse(
        generate(),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "X-Accel-Buffering": "no"  # Disable Nginx buffering for real‑time push
        }
    )

Key design of the Pipe layer:

# Model routing table: Open WebUI supports multiple back‑ends simultaneously
MODEL_ROUTES = {
    "ollama/*": OllamaHandler,   # Local Ollama models
    "openai/*": OpenAIHandler,    # OpenAI API
    "anthropic/*": AnthropicHandler,  # Anthropic Claude
    "custom/*": CustomPipeHandler,   # User‑defined Pipe
}

class PipeRouter:
    async def route(self, model_id: str) -> BaseHandler:
        for pattern, handler_class in MODEL_ROUTES.items():
            if fnmatch(model_id, pattern):
                config = await get_model_config(model_id)
                return handler_class(config)
        raise ModelNotFoundError(model_id)

Layer 3: Pipe Plugin System

Pipe is the most underrated feature of Open WebUI—it lets you intercept and rewrite any inbound or outbound message with a Python function.

Analogy: Pipe is like Express.js middleware, but for AI conversation streams.

# A complete Pipe example: automatically inject the current time into every message
from pydantic import BaseModel
from typing import Optional, Generator

class Pipe:
    class Valves(BaseModel):
        # Configurable via UI
        inject_time: bool = True
        time_format: str = "%Y-%m-%d %H:%M"

    def __init__(self):
        self.valves = self.Valves()

    def pipe(self, body: dict, __user__: Optional[dict] = None, __event_emitter__ = None) -> Generator:
        """
        body: full OpenAI‑style request payload
        returns: modified response stream
        """
        if self.valves.inject_time:
            now = datetime.now().strftime(self.valves.time_format)
            # Inject time into system prompt
            if body.get("messages") and body["messages"][0]["role"] == "system":
                body["messages"][0]["content"] += f"

Current time: {now}"
            else:
                body["messages"].insert(0, {
                    "role": "system",
                    "content": f"Current time: {now}"
                })
        # Call downstream model (passthrough modified request)
        response = requests.post(self.base_url + "/chat/completions", json=body, stream=True)
        # Stream response back
        for chunk in response.iter_lines():
            if chunk:
                yield chunk.decode()

Difference between Pipe and Function (both exist in Open WebUI):

Pipe (pipeline):
  ├── Intercepts the entire chat request
  ├── Can modify system prompt, messages, model parameters
  ├── Returns the full streaming response
  └── Appears as a new model in the model list (acts as a proxy)

Function (tool):
  ├── Triggers only when the model decides to call it
  ├── Receives tool_call parameters and returns a result
  ├── Result is injected back into the ongoing conversation
  └── Provides external capabilities (search, compute, DB lookup)

Takeaway: use Pipe when you need full‑process control (modify prompts, switch models, log), and use Function when you want to add external abilities (search, compute, DB access) to the model.

Layer 4: Storage Layer

Open WebUI defaults to SQLite; production deployments should switch to PostgreSQL—not for raw performance but to avoid concurrency lock issues.

# Core data model: chat history storage structure
# backend/open_webui/models/chats.py
class Chat(Base):
    __tablename__ = "chat"
    id = Column(String, primary_key=True)
    user_id = Column(String, ForeignKey("user.id"), nullable=False)
    title = Column(Text)
    # Store full messages array as JSON – enables "edit history" feature
    chat = Column(JSON)
    created_at = Column(BigInteger)
    updated_at = Column(BigInteger)
    archived = Column(Boolean, default=False)
    share_id = Column(Text, unique=True, nullable=True)

SQLite → PostgreSQL migration pitfalls and correct procedure:

# ❌ Simply changing DATABASE_URL in docker‑compose.yml and restarting wipes data
# → SQLite file is not auto‑migrated

# ✅ Proper migration steps
# 1. Backup existing data
docker exec open-webui cp /app/backend/data/webui.db /app/backend/data/webui.db.bak

# 2. Export data (Open WebUI provides an export UI)
# Admin > Database > Export

# 3. Start PostgreSQL and set env var
DATABASE_URL=postgresql://user:pass@postgres:5432/openwebui

# 4. Restart service (Alembic migrations run automatically)
docker-compose restart

# 5. Import previously exported data
# Admin > Database > Import

Vector database (RAG) support:

# Open WebUI’s RAG storage supports multiple vector stores, configurable via .env
# Default: Chroma (embedded, zero‑config)
VECTOR_DB=chroma

# Production recommendation: Qdrant (better performance, persistence)
VECTOR_DB=qdrant
QDRANT_URI=http://qdrant:6333

# Enterprise option: Milvus
VECTOR_DB=milvus
MILVUS_URI=http://milvus:19530

Layer 5: Model Connection Layer

Open WebUI uses a "unified interface + adapter" pattern to connect all models—internally everything speaks OpenAI format, and adapters translate to the specific backend.

# Base adapter class
class BaseModelHandler:
    async def stream(self, body: ChatCompletionRequest) -> AsyncGenerator:
        raise NotImplementedError

# Ollama adapter: translate OpenAI format to Ollama format
class OllamaHandler(BaseModelHandler):
    async def stream(self, body: ChatCompletionRequest) -> AsyncGenerator:
        # OpenAI → Ollama format
        ollama_body = {
            "model": body.model.replace("ollama/", ""),  # strip prefix
            "messages": body.messages,
            "stream": True,
            "options": {
                "temperature": body.temperature,
                "num_predict": body.max_tokens,
            },
        }
        async with aiohttp.ClientSession() as session:
            async with session.post(f"{self.ollama_base_url}/api/chat", json=ollama_body) as response:
                async for line in response.content:
                    if line:
                        ollama_chunk = json.loads(line)
                        # Ollama → OpenAI unified output
                        yield self._to_openai_format(ollama_chunk)

    def _to_openai_format(self, ollama_chunk: dict) -> ChatCompletionChunk:
        return ChatCompletionChunk(
            id=f"chatcmpl-{uuid4()}",
            choices=[{
                "delta": {"content": ollama_chunk.get("message", {}).get("content", "")},
                "finish_reason": "stop" if ollama_chunk.get("done") else None,
            }],
        )

Benefit: the front‑end only needs to consume a unified OpenAI‑style SSE stream, regardless of the underlying model.

Production Deployment: Three Must‑Change Settings

1. Disable Open Registration, Switch to Invite‑Only

# .env configuration
ENABLE_SIGNUP=false           # Disable public sign‑up
ENABLE_LOGIN_FORM=true        # Keep login form
DEFAULT_USER_ROLE=pending     # New users default to pending, require admin approval

# Or enable OAuth for corporate SSO
ENABLE_OAUTH_SIGNUP=true
OAUTH_PROVIDER_NAME=Company SSO
OPENID_PROVIDER_URL=https://sso.company.com/.well-known/openid-configuration
OAUTH_CLIENT_ID=your-client-id
OAUTH_CLIENT_SECRET=your-secret

2. Rate‑Limit to Prevent a Single User from Exhausting API Quota

# Open WebUI has no built‑in rate limiting; add it at the Nginx layer
# nginx.conf
limit_req_zone $binary_remote_addr zone=ai_api:10m rate=10r/m;

location /api/chat/completions {
    limit_req zone=ai_api burst=5 nodelay;
    limit_req_status 429;
    proxy_pass http://open-webui:8080;
}

3. File Upload Size and Format Restrictions

# Default has no limits; in production you must cap uploads to avoid abuse
MAX_UPLOAD_SIZE=50mb           # Max 50 MB per file
ALLOWED_UPLOAD_EXTENSIONS=pdf,txt,md,docx,xlsx,png,jpg,jpeg,gif,webp

Common Pitfalls

Pitfall 1: Streaming response becomes a single batch after Nginx reverse proxy.

# ❌ Default Nginx buffers the response; users wait for the whole model output
proxy_pass http://open-webui:8080;

# ✅ Disable buffering for true streaming
proxy_pass http://open-webui:8080;
proxy_buffering off;          # Turn off proxy buffering
proxy_cache off;             # Disable cache
proxy_read_timeout 300s;     # Long timeout to avoid cutting off long dialogs

Pitfall 2: Multiple users share a single API key, making usage tracking impossible.

# ❌ All users use the same OpenAI key
OPENAI_API_KEY=sk-proj-xxx

# ✅ Use Open WebUI’s API‑Key management to assign each user a dedicated key
# Admin > Settings > API Keys > Create a key per user
# This allows per‑key usage tracking in the OpenAI dashboard

Pitfall 3: RAG fails to retrieve answers after document upload.

Typical cause: chunk size is too large, leading to imprecise embeddings.

# ❌ Default chunk size 1500 tokens is too big for dense technical docs
CHUNK_SIZE=1500
CHUNK_OVERLAP=100

# ✅ For technical docs/code, reduce chunk size
CHUNK_SIZE=512          # Smaller chunks = more precise retrieval
CHUNK_OVERLAP=50        # Small overlap to keep context continuity

# Also adjust recall count
RAG_TOP_K=5            # Default 3; increase for better recall

Pitfall 4: Docker restart causes loss of chat history.

# ❌ No data volume mounted → data lost on container restart
services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main

# ✅ Mount /app/backend/data to persist all data
services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    volumes:
      - open-webui:/app/backend/data  # Persist data
volumes:
  open-webui:

Pitfall 5: Custom Pipe does not appear in the model list after implementation.

# ❌ Pipe class name not exactly "Pipe" or method name not "pipe"
class MyCustomPipe:          # Wrong: class must be named Pipe
    def run(self, body):     # Wrong: method must be named pipe
        ...

# ✅ Follow naming conventions
class Pipe:
    def pipe(self, body: dict, **kwargs):
        ...

Checklist: Production Deployment

Pre‑deployment checks:
- [ ] Data volume mounted (/app/backend/data)
- [ ] Public registration disabled (ENABLE_SIGNUP=false)
- [ ] HTTPS configured (Nginx + Let’s Encrypt)
- [ ] Nginx proxy buffering disabled (proxy_buffering off)
- [ ] File upload size limits set
- [ ] API keys assigned per user for usage tracking

RAG configuration:
- [ ] CHUNK_SIZE tuned for document type (technical docs → 512)
- [ ] Vector store switched to Qdrant or Milvus for >1000 docs
- [ ] RAG_TOP_K adjusted for retrieval quality

Multi‑model setup:
- [ ] Each model endpoint reachable
- [ ] Model permissions set per user role
- [ ] Model aliases configured to hide internal URLs

Monitoring:
- [ ] Nginx access logs configured
- [ ] Disk space alerts for uploads
- [ ] Regular database backups scheduled

Final note: Open WebUI’s core design philosophy is "everything is a Pipe"—once you understand the Pipe mechanism, you can extend any capability without touching the source code.

Follow me, James, for more practical AI‑era insights.