HarmonyOS 7 Local LLM Integration: Capabilities, Benchmarks & Engineering Guide

This article shares practical experience integrating a local large language model (LLM) into a HarmonyOS 7 application (project "BookDive"). It covers the capabilities of on-device LLMs, real-world performance measurements, a recommended architecture, step-by-step integration details, critical performance optimizations, and lessons learned from encountered pitfalls.

What Local LLMs Can Do in Apps

The author identifies several suitable tasks for a small on-device model:

Short Q&A and test entry : Verify model loading, tokenizer, prompt template, inference speed, streaming output, and error handling.

Narrative feedback generation : Each turn feeds only the current scene, relevant retrieved chunks, world state, and user action to generate a short narrative response.

Lightweight adjudication assistance : Judge whether an action advances the plot, violates world consistency, or triggers relationship/quest changes. The MVP phase still recommends rule-based fallbacks.

Summarization and rewriting : Produce short summaries of chapters or chunks, polish event sentences, generate scene titles — but input length must be strictly limited.

Character voice completion : Generate a few lines of dialogue under existing character cards and scene constraints; not suitable for open-ended companion chat.

Offline privacy : The novel text stays on the device; the app does not upload text to cloud models.

The core insight: local LLM value is not "reading the whole book on the phone" but performing generation, judgment, and polishing on retrieved local context. Long-novel understanding still relies on local chunking, indexing, world state, and a plot state machine.

Current Benchmark Results

Test device: HUAWEI Mate 60 Pro . Model: Qwen2.5-0.5B-Instruct in GGUF Q4_K_M quantization, runtime: native C++ + llama.cpp.

After CPU build optimizations, short Q&A latency dropped from 20–30 seconds to about 1–2 seconds, with decode throughput around 40 tokens/second.

Stage | Performance | Reason
------|-------------|-------
Initial native integration | Simple Q&A 20-30s, some scenes 40s+ | Debug native build missed CPU optimizations; prompt and output also heavy
Async & streaming output | Perceived improvement, UI no longer freezes, but total latency still high | Inference moved to native background thread; ArkTS only handles state and rendering
CPU build optimization (O3, LTO, KleidiAI, ARM dotprod/fp16) | Short Q&A ~1-2s, decode ~40 tok/s | O3, LTO, KleidiAI, ARM dotprod/fp16 paths enabled
Quality guardrails applied | Hallucinations reduced, but 0.5B still unstable | Low temperature / greedy, short prompt, quality guard, template fallback

Key takeaway : Speed alone does not equal product readiness. The 0.5B model proves the on-device generation pipeline works, but comprehension and stability are limited. For interactive narrative, it serves as technical validation and low-end fallback; product candidates should evaluate from 1.5B/2B upward.

Recommended Architecture

The current solution uses ArkTS for the application layer and Native C++ for the inference layer , with llama.cpp handling GGUF model loading and token generation. The business layer does not stuff the entire novel into the model; instead, it controls context via local parsing, chunking, retrieval, and a state machine.

Core Module Responsibilities

Module | Responsibility | Current Implementation
-------|----------------|--------------------
Manifest | Declare model, runtime, format, tokenizer, generation parameters | local_llm_validation_manifest.json
ArkTS Adapter | Build prompt, control timeout, select fallback, update UI | LocalLLMGeneratorAdapter.v0
Model file preparation | Copy model from HAP rawfile to app sandbox on first run | Avoids reading large file from rawfile repeatedly
Native Runner | Load GGUF, create context, inference, streaming callbacks | libbookdive_llm.so
Fallback | Provide readable result on model failure or timeout | TinyBookNarrativeLM.v1 / template generation

Integration Steps

1. Choose Model and Format

MVP phase prioritizes small models and low-risk formats. Current validation uses 0.5B Q4 quantized GGUF to confirm the on-device inference, memory, thermal, streaming, and fallback pipeline — not final quality.

{
  "provider": "LocalLLMGeneratorAdapter.v0",
  "selectedModel": {
    "id": "Qwen/Qwen2.5-0.5B-Instruct",
    "parameterCount": "0.49B"
  },
  "runtime": {
    "selected": "NativeCppGGUF",
    "nativeEntry": "libbookdive_llm.so"
  },
  "format": {
    "preferred": "GGUF-Q4_K_M",
    "preferredModelFile": "models/qwen2.5-0.5b-instruct-q4_k_m.gguf"
  }
}

2. Bundle Model in HAP for POC

To reduce variables, the POC places the GGUF model in the HAP's resources/rawfile/models. This verifies a fully local chain with no network, no download, no account. However, this is not a long-term solution: when models grow to 1.5B/2B, the HAP size balloons. The better approach: ship base functionality in the initial package, let users import or download model packages to the sandbox later — still without uploading novel text.

3. ArkTS Adapter Wrapper

ArkTS does not deal with llama.cpp internals; it only constructs prompts, sets parameters, receives streaming tokens, and handles fallbacks.

const rawResult = await bookdiveLlm.generateStream(
  modelPath,
  prompt,
  {
    maxNewTokens: 24,
    timeoutMs: 9000,
    temperature: 0.35,
    topP: 0.75,
    contextTokens: 384,
    nThreads: 4,
    nThreadsBatch: 4,
    batchSize: 128,
    ubatchSize: 64
  },
  (token: string) => {
    receiveStreamToken(token);
  }
);

4. Native C++ Layer Interfacing llama.cpp

The native layer exposes generate, generateAsync, generateStream, and prepareModelFileAsync. Heavy inference must run in native async work or background threads, never blocking the ArkUI thread.

// Simplified NAPI contract
generateAsync(modelPath, prompt, options) => Promise<string>
generateStream(modelPath, prompt, options, onToken) => Promise<string>

// Returned JSON
{
  "ok": true,
  "text": "...",
  "provider": "NativeGGUFRunner.llama.cpp",
  "promptTokens": 73,
  "decodedTokens": 16,
  "firstTokenMs": 938,
  "tokensPerSecond": 40.5
}

5. Prompt Feeds Only Necessary Information

The project's local LLM receives four categories per turn:

Current scene: title, location, characters, goal

Relevant chunks: few segment summaries retrieved from local index

World state: turn count, quests, relationships, adjudication tags

User action: the user's input for this turn

System: You are BookDive's local narrative generator. Continue only based on given world info.
User:
Scene: Night at the Inn
Location: Inn
Characters: Protagonist, Innkeeper
Goal: Figure out footsteps outside
Clue: Innkeeper hears footsteps outside
State: Turn 3, quest progressing, relationship 0
User action: I quietly ask the innkeeper what happened
Requirement: One narrative feedback; no rule explanations.

Performance Optimization Key Points

The performance leap came not from a stronger model but from enabling llama.cpp optimized paths on HarmonyOS ARM CPUs.

set(GGML_LTO ON CACHE BOOL "" FORCE)
set(GGML_CPU_ARM_ARCH "armv8.2-a+dotprod+fp16" CACHE STRING "" FORCE)
set(GGML_CPU_KLEIDIAI ON CACHE BOOL "" FORCE)

set(BOOKDIVE_ARM_CPU_FLAGS "-fvectorize -ffp-model=fast -fno-finite-math-only")
set(CMAKE_C_FLAGS_DEBUG "-O3 -DNDEBUG -g0" CACHE STRING "" FORCE)
set(CMAKE_CXX_FLAGS_DEBUG "-O3 -DNDEBUG -g0" CACHE STRING "" FORCE)

Optimization | Effect | Caveats
-------------|--------|--------
O3 / DNDEBUG | Avoids Debug builds actually running O0 — a major pitfall for mobile CPU inference | Verify build.ninja to confirm flags take effect
GGML_CPU_KLEIDIAI | Enables ARM CPU optimized kernels | Must confirm target device ABI and instruction capabilities; don't blindly target too high an architecture
LTO | Link-time optimization, reduces inference path overhead | Increases build time
Small ctx / short output | Significantly reduces prefill and decode time | Real narrative cannot be compressed indefinitely without quality loss
Context reuse | Avoids rebuilding llama context every turn | Must clear KV cache each turn to prevent context bleed
Streaming output | Improves perceived latency, lets user see text early | Requires handling token callback throttling and UI refresh rate

Pitfalls Encountered

Slowness Isn't Always Model Size — It May Be Unoptimized Build

The 0.5B model initially took 20–30 seconds, easily misjudged as "phone CPU can't run LLM". Enabling O3, KleidiAI, LTO brought short Q&A down to seconds.

0.5B Hallucinates — Quality Guardrails Are Mandatory

Small models under short prompts and random sampling tend to output irrelevant boilerplate. For example, asking "What can you do?" might yield unrelated archaic sentences. Mitigations:

Test entry uses temperature=0 or greedy decoding.

Narrative generation uses low temperature and smaller topP.

Prompt explicitly states "only answer the current input".

Quality guard for fixed commands, greetings, capability questions.

Fallback to template / TinyBookNarrativeLM on model failure or off-topic output.

Real Token Counts Must Come from Native Tokenizer

ArkTS-side token estimation is only a budget hint, not a safety boundary. A crash occurred when a growing prompt caused actual token count to exceed n_batch, triggering ggml_abort inside llama.cpp. Fix: perform real tokenization in native layer and decode in batches.

bool DecodePromptInChunks(ctx, tokens, batchSize, error) {
  for (offset = 0; offset < tokens.size(); offset += batchSize) {
    current = min(batchSize, tokens.size() - offset);
    batch = llama_batch_get_one(tokens.data() + offset, current);
    if (llama_decode(ctx, batch) != 0) return false;
  }
  return true;
}

Don't Put Heavy Work on the UI Thread

Model copying, loading, tokenization, prefill, decode are all heavy. ArkTS pages only manage state and render; local TXT parsing runs in a worker, LLM inference runs in native async work.

HAP-Bundled Models Only for Validation

HAP-bundled 0.5B model enables quick pipeline validation, but package size becomes uncontrollable as models grow. For 1.5B/2B+, implement model package management: download, verify, sandbox storage, version switching, low-end device fallback.