Run LLMs on Old Android Phones with OlliteRT: OpenAI-Compatible Local Server

Turn an Android Phone into a Local LLM Server

OlliteRT is an Android application that runs supported language models on the device and turns the phone into an OpenAI API-compatible inference server. Other devices on the same LAN can send requests to the phone's API endpoint (e.g., http://phone-lan-ip:8000/v1) using any OpenAI-compatible client such as Open WebUI, Python scripts, or curl.

Quick Start Steps

Download and install the APK.

In the app, download or import a supported model.

Tap the model card's start button to launch the server.

In your client, enter the service address shown on the status page.

The phone and client devices must be able to reach each other on the local network.

Core Features

Model management : Download from Hugging Face, import local .litertlm files, or add custom model sources via JSON or URL.

Multimodal capabilities : For supported models, enables vision, audio, reasoning traces, and streaming output; tool calling is experimental.

Observability : Request/response logs with search and filter, plus model and service status monitoring.

Runtime configuration : Adjust inference parameters, CPU/GPU acceleration, idle model unloading, listening scope, and client IP access rules.

Integrations : OpenAI-compatible API, Prometheus metrics endpoint, and Home Assistant REST API.

Suitable Scenarios

Personal local experimentation : Test on-device model behavior and compare small models.

Connect existing AI clients : Point Open WebUI or custom Python tools at the phone's API to evaluate local inference latency and quality.

Home automation exploration : Use the Home Assistant integration for local AI experiments (voice capabilities require additional HA configuration).

Repurpose idle devices : A spare phone can serve as a lightweight inference node for demos or testing, though sustained loads cause heat, battery drain, and aging.

Key Limitations

Requires Android 12+, arm64-v8a architecture, minimum 6 GB RAM (8 GB+ recommended for multimodal models).

Only .litertlm format is supported; GGUF is not supported.

Only one model can be loaded at a time; requests are processed sequentially.

Tool calling depends on the model's own capability and remains experimental.

Some inference features may differ from desktop frameworks due to runtime constraints.

These constraints make OlliteRT best suited for edge experimentation, lightweight services, and development validation rather than production workloads.

Security Considerations

Local inference keeps data on-device, but exposing the API over LAN requires access control. OlliteRT provides Bearer Token authentication, client IP allow/deny lists, and configurable listening interfaces. The project states it collects no telemetry or analytics; details are in the repository's privacy policy and security guide.

Repository and Resources

Open-source repository: https://github.com/NightMean/OlliteRT

The repository contains source code, installation instructions, and the latest model support list.