Zero‑Cost Inference on Edge: A Qwen 3.8‑27B‑Powered Harness for Local‑First Agents

Background and Motivation

Typical harnesses assume access to large‑model APIs (e.g., Fable 5, Sol, Kimi K3). Perplexity’s Portable Computer instead runs the entire stack locally, keeping model, framework, dialogue, and trace on the user’s device. This eliminates inference fees and prevents private data from leaving the device.

Design of a Local‑First Harness

Because off‑the‑shelf harnesses such as Pi and Hermes are not optimized for edge models, Perplexity built a custom harness around three principles.

Context efficiency : Although Qwen 3.8 27B offers a 260 k‑token window, experiments showed performance drops after ~100 k tokens. The harness therefore contains only a minimal system and a core toolset; all other capabilities are modular skills loaded on demand.

Connector as CLI tool : Common connectors (Gmail, GitHub, Outlook, Calendar) are exposed as lightweight command‑line utilities rather than heavyweight MCP servers, saving context space.

Self‑verification : Hooks monitor trajectory health and trigger model‑self‑verification or external verification when problems arise, improving result quality.

Sandboxed execution : Tools run inside an OS‑level sandbox with strict policies on processes, file‑system paths, and network access. If the sandbox is unavailable, the framework disables tool execution entirely.

Benchmark Methodology

Perplexity evaluated the harness on three public benchmarks and an internal “Local Knowledge Work Bench” using the same Qwen 3.8 27B model on an NVIDIA DGX Spark.

Web search : 1 266 BrowseComp tasks. Portable Computer achieved 66.7 % accuracy, compared with 50.2 % (Pi) and 43.9 % (Hermes). Average runtime was 402.1 s and token usage 852 k, 61 % faster and 16 % fewer tokens than Hermes.

Multimodal document understanding : ParseBench‑100 (100 tasks across charts, layouts, tables, text, format). Portable Computer scored 65.1 %, far above Hermes (34.6 %) and Pi (13.9 %). Average time per task was 60.6 s with 20.1 k tokens, again the lowest among the three.

Terminal Bench 2.1 (89 coding tasks). The local model lagged behind the frontier Claude Opus 5, prompting an optional “advisor” upgrade.

Advisor Upgrade

When a task exceeds the edge model’s capability, the coordinator can forward a concise context to a stronger cloud model (Claude Opus 5). The upgrade raises Portable Computer’s score from 59.6 % to 73.0 % at an estimated API cost of $0.415 per deployment, compared with $0.65 for running Claude Opus 5 alone.

Post‑Training (PPLX 27B)

Perplexity built a reinforcement‑learning pipeline: first reject‑sampling fine‑tuning, then RL‑HF on synthetic “knowledge‑work” tasks derived from real‑world use cases. The resulting PPLX 27B model improved Portable Computer’s benchmark score to 85.4 % (up from 82.6 % for the base Qwen 3.8 27B) while increasing token consumption and runtime to 250 s.

Conclusions

Strong open‑source models combined with purpose‑built local hardware and a minimalist, sandboxed harness can perform knowledge‑work at near‑zero inference cost while preserving privacy. However, high‑difficulty coding tasks still benefit from occasional cloud‑advisor upgrades, and further advances in model capability and edge hardware are needed.