Debug HarmonyOS 7 XiaoYi Skills: From Platform Evaluation to On-Device Testing
This article details a systematic debugging methodology for HarmonyOS 7 XiaoYi skills, showing how to trace discrepancies from platform evaluation reports through tool integration to on-device testing by comparing fixed inputs, expected outputs, and actual results at each execution stage to locate the earliest point of failure.
Introduction: Passing Tests but Wrong Output
The author describes a case where the first evaluation report showed three test groups all passing (9/9, 4/4, 3/3 assertions), yet the downloaded skill file incorrectly set the meeting date to a default value, used an array for original evidence, and a different field name for suggestions. After revision, the report passed again but the actual output still contained meetingDate in the action's missingFields. This demonstrates that summary pass rates alone cannot reveal downstream issues when the program later reads results.
Debugging Methodology: Compare Along the Execution Chain
Troubleshooting requires placing original input, expected output, and actual output side by side, then walking the execution order to find the earliest divergence. The author outlines the comparison sequence:
Text structuring stage: compare category, assignee, and date.
After tool integration: verify call parameters, HTTP responses, and database contents.
When saving to device: continue comparing notes.
Only after confirming what the previous stage provided can you determine where the next stage altered the content, avoiding simultaneous changes to rules and services when the final answer is wrong.
Fixed Test Cases: samples/07-verification/evaluation-cases.json
The file contains 15 fixed test cases (N01–N15) covering text structuring, queries, confirmation changes, cancellation, response loss, and second-target failure. Each case specifies: id (e.g., N01, N02) risk description input text (Chinese) expected outcome description platformStatus, platformActual, deviceStatus, deviceActual (initially all NOT_RUN / null)
Example N01: input "蓝桥,会议日期2026-10-04。林舟在10月8日前确认接口字段。请只整理,不保存。" expects actions containing 林舟/2026-10-08 and original evidence; no write tool calls.
Example N02: input "蓝桥,会议日期2026-10-04。10月8日前确认接口字段,负责人还没有确定。请保存。" expects owner=null, missingFields contains owner, skill asks for assignee and does not create proposal.
The author emphasizes keeping original inputs unchanged when modifying rules, so that specific deviations can be tracked. Changing test text to make new results pass loses the original problem.
Platform Evaluation Reports: First Run vs. Revised
In the HarmonyOS developer platform, open the existing meeting-actions project, enter the generation session, and open the evaluation report. The first run showed with-skill 100% (16/16 assertions) and without-skill 68%±39. The third without-skill group also passed completely, so one cannot conclude that adding the skill improved every date judgment.
Expanding assertions reveals the first report checked category, missing assignee, and relative date, but did not catch the teaching-value backfill, source array, and suggestions.content field differences from later reading conventions. The initial zip contained only SKILL.md, no module.json. Validation, packaging, upload, security scan, and BENIGN are UI-stage results; missing files are not eliminated by stage passes.
The revised session added module.json (version 0.2.0, empty dependencies), changed source to string, and used title for suggestions and cancellations. These changes are verifiable from exported files, but byte-different packages with the same version must still be compared by content.
Locating the Next Fix from Real Field Deviations
Opening the revised first actual output ( samples/02-platform-revised/01-actual.json) shows root meeting metadata all null, action owner "张磊", dueDate null. The missingFields array contains both dueDate and meetingDate. Current business convention allows only owner and dueDate in action missingFields; meeting date is root information and must not be mixed in. The SKILL.md lacks a complete restriction on missingFields values, and conflict rules still allow clarification content there. The next adjustment should explicitly limit missingFields to owner and dueDate, place meeting date missing and conflict prompts outside JSON, and re-verify with the same input.
Tool Integration: Trace the First Real Request Difference
After cloud tool binding, comparison extends to actual calls. Using N08 (query blue-bridge open actions), check whether the target skill triggers, then whether the tool is called with parameters blue-bridge and open, and whether the service receives those values. Only after service request is correct compare HTTP response actions with final list.
If the platform emits no request, check real resource dependencies, address reachability, and access conditions. If the model gives correct parameters but the service receives different ones, inspect tool input mapping. If the model selects all but the service returns completed actions for all, that is the actual parameter result; do not change query code to always return only open actions.
A local HTTP check script ( examples/demo-service/test_service.py) is provided for service behavior verification. It runs real loopback HTTP and SQLite tests for queries, approvals, concurrent commits, response loss, restart, and expiry. Example checks:
status, r = request('GET', '/api/actions?projectId=blue-bridge&status=open')
check('Q01 正常查询两个未完成行动', status == 200 and [i['id'] for i in r['items']] == ['A-001', 'A-002'])
status, r = request('GET', '/api/actions?projectId=empty-project&status=open')
check('Q02 合法空结果', status == 200 and r['items'] == [])The script starts a local server, runs 26 checks (Q01–Q05, W01–W12, F01–F06, P01, S01–S02), and writes results.json and results.md. The final manifest notes platform, cloud plugin, memo plugin, and device as NOT_RUN / NOT_BOUND.
On-Device Testing Setup
Open the real-device test entry in the Vibe project. The page shows two entries: Claw (install prompt) and Main Dialog (requires HarmonyOS 7, XiaoYi 11.7.x or higher, whitelist user). Main Dialog test period is 15 days, sandbox default off. No test publish, skill install, or whitelist modification has been done yet.
Alternative flow uses test user groups: admin selects groups (max 100 groups, 100 users each, max 10 groups per publish, 15-day validity). Current Vibe whitelist differs from this flow; do not assume whitelist is configured just because group template exists.
A device record template ( samples/07-verification/device-record-template.md) captures identity, skill package, tools, device/service conditions, and per-case records (fixed case ID, original input, triggered skill, tool call sequence and parameters, HTTP request/response IDs, proposalId/operationId, user approve/cancel time, DB/result page changes, device note title/content/count, final feedback, deviation, first deviation stage, conclusion, screenshots). Template items remain 待填写 until actually executed.
On-Device Comparison: Same Device, Step-by-Step
When conditions are met, start with N01 (text-only structuring), save original input, full structure, and no write call records. Then N02 (missing assignee), confirm null and follow-up, observe stop before proposal. Keep meeting date fixed in input, not replaced by device today. Each case saves current skill and resource identity.
After query tools are connected, use normal and empty result inputs, compare tool parameters, HTTP responses, and final lists. After create/commit integration, check pre-confirm cancellation, content changes, response loss. Content change requires seeing new proposal and re-approval; button display alone is insufficient. Real operation records link preview, approval, and write to same list.
Device tool testing opens real Memo on device, compares title, content, count. After tool success, still verify 陈禾's pending date, suggestions, cancellations are retained, original text complete. Pre-confirm cancellation must not produce new save call; after assignee change, observe new preview then confirm. Tool failure and timeout handled by explicit result and pending verification respectively.
Summary
Two evaluation rounds identified a concrete issue: action missingFields incorrectly includes meetingDate. Fix should target this field constraint first, then re-verify with same input — not substitute another report's percentage. After tool and device integration, continue the same method: align parameters, service results, and notes sequentially to find the earliest deviation from expectation. All 15 independent inputs must run in environments with corresponding identities, versions, and resources. Record business success, target failure, and unknown results separately; cancellation feedback must match already-occurred writes.
Reference Files
15 Fixed Inputs and Expectations
Path: samples/07-verification/evaluation-cases.json. Contains N01–N15 with full input, expected, and actual columns. Reuse inputs with original dates, names, operation scopes. Cases needing query, approval, or device tools require satisfying those conditions first.
Platform and Device Operation Record Template
Path: samples/07-verification/device-record-template.md. Captures test identity/version, per-case fields, platform evaluation, and publish condition checks. Template items stay as placeholders until actual execution.
Platform Raw Output with Field Deviation
Path: samples/02-platform-revised/01-actual.json. Shows root meeting info null and action missingFields containing both dueDate and meetingDate (latter out of convention). Retain this original output for before/after rule adjustment comparison.
Local HTTP and Database Check Program
Path: examples/demo-service/test_service.py. Real loopback HTTP + SQLite tests for query, approval, concurrent commit, response loss, restart, expiry. Deletes test-results/actions.sqlite* on start; preserve history by copying directory first.
Local Service Historical Check Results
Path: examples/demo-service/test-results/results.md. 26 checks from 2026-10-04, all PASS. Platform, cloud plugin, memo plugin, device not executed.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
