OlliteRT: Turn Old Android Phones into Local LLM API Servers

What is OlliteRT

OlliteRT is an open-source Android application (Apache 2.0, Kotlin) that turns an Android phone into a local large language model server. It is built on Google's LiteRT-LM runtime, originally derived from the AI Edge Gallery demo app, and repackaged by developer NightMean as a proper server application. The app exposes an OpenAI-compatible REST API on the local network (default port 8000), allowing any client that accepts an OpenAI base URL — such as Open WebUI, Home Assistant, Python scripts, or curl — to send inference requests to models running entirely on the phone. No cloud services or API keys are required; data never leaves the local network.

Capabilities and Supported Models

The app includes a built-in model catalog with seven pre-configured models. The officially recommended model is Gemma 4 E2B (2.4 GB), which runs on phones with 8 GB RAM and supports text, image, and voice input, plus a "thinking" mode and tool calling (experimental). Smaller options include Qwen 2.5 1.5B and DeepSeek-R1 1.5B. All models use the proprietary .litertlm format; GGUF is not supported. Users can also import custom .litertlm files or add external model sources via a JSON file or URL.

A built-in benchmark tool lets users measure decoding speed before committing to a download. Official screenshots show Gemma 4 E2B achieving 29.9 tokens/s with a 4-second model load time. Context windows vary: Gemma 4 series offer 32K tokens, while Gemma 3n and the smaller models provide only 4K.

Configuration and Security

Per-model settings include temperature, max output tokens, choice of GPU or CPU execution, and an auto-unload timeout. Security features cover network scope (LAN-only or broader), client IP allowlists, and optional Bearer token authentication.

Observability and Integration

The status page streams real-time metrics: uptime, request count, token throughput, decode speed, and success rate. For advanced monitoring, OlliteRT exposes 29 Prometheus metrics compatible with Grafana dashboards. Home Assistant users can integrate via a REST API to query status and control model loading remotely. Full request/response logging with search, filtering, and JSON highlighting is provided for debugging.

API compatibility extends beyond OpenAI: the server also implements Anthropic's Messages endpoint, a rarity in lightweight local-serving tools.

Quick Start Guide

Install the APK from the GitHub Releases page. Requirements: Android 12+, arm64 architecture (most post-2017 devices), minimum 6 GB RAM (8 GB+ recommended for multimodal models).

Download a model from the in-app list (Gemma 4 E2B suggested) or import a .litertlm file manually.

Start the server on the model card. The status page displays the API endpoint (e.g., http://192.168.1.176:8000/v1).

Connect a client by entering three values: the API base URL, an API key (any placeholder if auth is disabled), and the exact model name (e.g., Gemma-4-E2B-it).

Integration examples:

Open WebUI: Add an OpenAI connection in Settings → Connections; enable Function Calling → Native for tool support.

Home Assistant: Install the Custom Conversation integration via HACS; configure with the same three values. Enables both chat and speech-to-text.

Claude Code: Set two environment variables pointing to the phone's endpoint. Note: the author warns that phone context windows are too small for heavy tool-use workloads; treat as a compatibility test only.

Limitations and Considerations

Single-model, single-threaded: only one model can be loaded at a time; requests are queued sequentially — suitable for personal/home use, not production concurrency.

Format lock-in: only .litertlm models are supported; existing GGUF libraries cannot be used.

Tool calling is experimental and unreliable.

Thermal constraints: sustained inference heats the device. The README humorously advises against placing a running phone under a pillow; aged batteries in old phones may swell — monitor if running 24/7.

Performance ceiling: cannot replace cloud LLMs for speed or model scale, but adequate for local voice assistants or light experimentation.

Conclusion

At 5–10 W power draw versus 300 W+ for GPU servers, an idle phone becomes a cost-effective, always-on, privacy-preserving AI endpoint. While not a substitute for cloud-scale inference, OlliteRT makes it practical to run a local voice assistant (akin to XiaoDu) or experiment with on-device LLMs. The project is young but well-engineered, with thoughtful observability, security, and integration hooks.

Source code: https://github.com/NightMean/OlliteRT