Tagged articles
On-device Inference
12 articles · Page 1 of 1
Recursive team open-sources Flint-40B-A2B, a 40B-parameter MoE model optimized for local coding agents, developed in under four months using their Eureka Machine AI research system; it achieves over 125 tokens/sec on MacBook Pro and demonstrates autonomous research capabilities on tasks like circle packing, factory scheduling (26% improvement), and bike-sharing demand prediction (30.8% RMSLE reduction).
Automated ResearchCoding AgentEureka Machine
0 likes · 6 min read

OlliteRT lets you run large language models on Android phones with 6GB+ RAM, exposing an OpenAI-compatible API for network clients, supporting model management, multimodal inference, and Home Assistant integration, though limited to .litertlm format and single-model loading.
AndroidEdge ComputingLLM
0 likes · 8 min read


Geek LabsSep 28, 2026 · Artificial Intelligence
OpenSquilla, a 7K-star open-source AI agent, uses on-device routing to classify each conversation turn by complexity and dispatch it to the cheapest suitable model, achieving 9x cost reduction on 25 benchmark tasks while maintaining near-identical scores, plus adaptive reasoning, dynamic prompts, and pluggable providers.
AI agentCost OptimizationLLM
0 likes · 9 min read

OlliteRT is an open-source Android app that runs large language models locally via Google's LiteRT runtime, exposing an OpenAI-compatible API over LAN for lightweight always-on tasks like RSS summarization and smart home automation, with monitoring, logging, and security features.
AndroidGemmaLLM
0 likes · 11 min read

This article details integrating local LLMs into HarmonyOS apps, covering use cases like narrative generation and offline privacy, real-world benchmarks on Mate 60 Pro with Qwen2.5-0.5B, architecture using ArkTS and llama.cpp, performance optimizations via O3/LTO/KleidiAI, and key pitfalls like token limits and UI threading.
GGUFHarmonyOSOn-device Inference
0 likes · 15 min read

nndeploy is an open‑source, high‑performance on‑device AI deployment framework that abstracts the repetitive “last‑mile” workflow into a visual drag‑and‑drop DAG, offering multi‑platform inference, optimization, and ready‑to‑use model configs, enabling developers to go from prototype to production in minutes.
AI DeploymentOn-device Inferenceedge AI
0 likes · 15 min read

NeuTTS‑Air, an open‑source on‑device text‑to‑speech model built on a 0.5B Qwen LLM and NeuCodec, reaches SOTA among open models, runs entirely on CPU, supports 3‑second voice cloning, and comes with a step‑by‑step tutorial for deployment on edge devices.
NeuCodecNeuTTS-AirOn-device Inference
0 likes · 5 min read

This article explains three traditional approaches to on‑device AI for mobile apps, compares their drawbacks, and introduces Sohu News' Hybrid AI Engine that unifies TensorFlow Lite and system‑level AI (HiAI) with concise code examples and future open‑source plans.
AndroidHybrid AI EngineOn-device Inference
0 likes · 11 min read

The article presents Alibaba’s open‑source MNN inference engine, demonstrating how quantization, operator fusion (including fused multi‑head attention, GroupNorm/SplitGeLU, Winograd convolutions), optimized GEMM and memory‑paging enable on‑device Stable Diffusion with 1‑second‑per‑step performance on Snapdragon 8 Gen3 and Apple M3 GPUs, and outlines future speed‑up directions.
AIMNNOn-device Inference
0 likes · 11 min read

The article outlines how Sohu’s Hybrid AI Engine enables on‑device deployment of a distilled GPT‑2 model by converting it to TensorFlow Lite, detailing the setup, customization with Keras, inference workflow, and core SDK calls, and argues that this approach offers fast, private, and cost‑effective AI for mobile devices despite typical LLM constraints.
GPT-2Hybrid AIKeras
0 likes · 9 min read

This article details the technical evolution of Kuaishou's short‑video recommendation pipeline, focusing on sequence re‑ranking, multi‑content mixing, and on‑device re‑ranking, and explains how transformer‑based models, generator‑evaluator frameworks, and reinforcement‑learning strategies are employed to maximize overall sequence value, user engagement, and revenue.
KuaishouOn-device InferenceRe-ranking
0 likes · 15 min read

This article introduces three traditional approaches for deploying machine‑learning models on mobile devices, analyzes their drawbacks, and presents a hybrid AI engine that combines TensorFlow Lite and system‑level HiAI to provide a unified, lightweight, and developer‑friendly on‑device image‑recognition solution, including code examples.
Android developmentHybrid AI EngineOn-device Inference
0 likes · 12 min read
