Run LLMs on 6GB Android Phones with OlliteRT: OpenAI-Compatible LAN API

OlliteRT is an open-source Android application (Apache-2.0) that brings local LLM inference to mobile devices using Google's LiteRT-LM runtime. The app leverages the phone's GPU or CPU to run models and serves an HTTP API on the local network that is compatible with OpenAI's chat/completions, completions, Responses, and audio transcription endpoints, as well as Anthropic's Messages API. Both streaming and non-streaming modes work.

Model Sources and Supported Models

Models can be obtained three ways: from a built-in list hosted on Hugging Face's LiteRT community, by importing existing .litertlm files stored on the device, or by adding a custom model source via a JSON file or URL (the app refreshes sources roughly every 24 hours). The built-in list currently highlights Gemma 4 E2B (recommended for most users) and E4B (for higher-end phones). Both are multimodal — handling images and audio — and support a "thinking" mode that emits a reasoning trace before the final answer. Gemma 4 also includes MTP (multi-token prediction) which accelerates token generation when enabled in the model settings.

API Compatibility and Integration

The API compatibility is a standout feature. Existing tools require minimal changes: Open WebUI connects by simply pointing to the phone's address; Python scripts using the official OpenAI SDK only need the base_url changed to http://<phone-IP>:8000/v1; curl works directly; Home Assistant has dedicated integration documentation. The author migrated several NAS cron jobs by updating their API endpoints and nothing else. Tool calling is experimental and works best with Gemma 4; smaller models may not reliably return structured output. Two modes are provided: schema injection (default) and a fallback prompt-based parsing approach.

Monitoring, Logging, and Observability

The app provides a real-time status page showing uptime, request count, input/output tokens, tokens-per-second, time-to-first-token, and success rate. A movable floating window lets the user glance at server health while the phone sits idle. The log page records every request and response with filtering by HTTP method, status code, and keyword search; JSON payloads are syntax-highlighted. Each log entry shows latency in milliseconds, streaming vs. non-streaming, thinking mode status, and estimated context usage (color-coded yellow/red as context fills). Additionally, a /metrics endpoint exposes 29 Prometheus-formatted metrics for Grafana dashboards. Home Assistant can also query a built-in REST API for remote status, model switching, and configuration changes. A built-in benchmark runs a suite of models sequentially to compare long-input processing speed, token generation speed, and load time — recommended for new devices.

Configuration and Operational Features

Each model stores its own inference parameters (temperature, top-k, top-p, max output length) which persist across restarts. The accelerator can be toggled between GPU (default) and CPU. Three practical switches are highlighted: Idle auto-unload — after a configurable period (e.g., 5 minutes) without requests, the model is evicted from memory and reloaded on the next request (adding a few seconds of latency); Boot auto-start — with a default model selected and "Start on Boot" enabled, the service launches automatically after reboot (battery optimization must be disabled for the app); Persistent notification — a foreground notification keeps the background service alive and provides one-tap stop and address-copy buttons.

Security Model

The API can be protected with a bearer token (requests without it receive 401). Client IP allowlists and bind-address restrictions are also configurable. The documentation explicitly warns against exposing the service to the public internet via port forwarding: it lacks HTTPS, token authentication is only a basic measure, and the intended deployment is strictly within a trusted LAN.

Setup and Hardware Requirements

Installation: download the APK from GitHub Releases, install, pick a model from the list, download it, tap "Start Server" on the model card. The server address appears on the status page (typically http://<phone-IP>:8000/v1). Requirements: Android 12+, arm64 architecture (virtually all post-2017 Android devices), minimum 6 GB RAM (8 GB+ recommended for multimodal models).

Limitations

Only one model can be loaded at a time; requests are queued and processed sequentially — unsuitable for multi-user serving.

Only .litertlm format is supported; GGUF models (common for llama.cpp/Ollama) are not compatible.

Token counts returned by the API are estimated (character count divided by four), accurate enough for English but skewed for code and multilingual text — not suitable for billing.

Importing a local model file copies it into the app's private directory, temporarily doubling storage usage; the original should be deleted after import.

Advanced features like logprobs and LoRA are unavailable due to the mobile runtime constraints.

Long-Running Deployment Advice

For 7×24 operation: keep the phone plugged in, remove the case for heat dissipation, place on a cool hard surface, and periodically check for overheating or battery swelling. Power draw during generation is roughly 5–10 W; idle draw is near zero — far below typical GPU servers (300+ W).

Overall Assessment

OlliteRT cannot replace a dedicated inference server due to single-model concurrency and request queuing. However, for fragmented, low-volume, privacy-sensitive workloads (RSS summarization, photo tagging, smart home command processing) where spinning up a full machine is overkill and data must not leave the premises, it fits well. The monitoring, logging, access control, and auto-start features make it viable as a set-and-forget appliance. The author's NAS automation tasks have run unattended since adoption. Spare Android phones meeting the specs are good candidates.

Open-source repository: https://github.com/NightMean/OlliteRT