AI Agent as New User: Automating Usability Testing with Doubao Work

Give It a Goal, Let It Explore

Developers often demo software smoothly, but first-time users quickly get stuck. To catch usability gaps before release, the author ran an experiment: use Doubao Work (豆包工作) as an AI "new user" to operate the AI writing desktop product "Lingxie Creation" (灵蟹创作) and produce an experience report.

The test used Doubao Work's standalone Mac desktop app in local-computer mode. The goal given to the agent was: "Use Lingxie Creation's Agent to conceive a novel, build the framework, and complete the first five chapters." The agent had to figure out how to start, track progress, and locate results solely by reading the UI and official documentation — no source-code access, no API shortcuts. The author added a rule: if the agent cannot distinguish a product flaw from its own operational limit, mark it "to verify."

Doubao Work drove the application: it viewed the interface, entered the requirement, obtained the story framework and chapter one, then asked the Agent to continue chapters two through five while maintaining continuity and updating memory files. During execution the agent hit a snag — a keyboard shortcut to open a folder failed — so it fell back to the existing workspace. The author notes these stumbling points are exactly where a real user would pause, signaling missing prompts or guidance.

The agent proved thorough: after five chapters were generated, it opened each chapter in the editor, copied the full text, cross-checked word-count methodology, and captured numerous screenshots. These artifacts became the evidence base for its final report.

The report separated items needing human verification from clear product issues, and ranked optimization suggestions by priority. This gave the team a concrete, itemized improvement checklist. The run consumed only 1.68% of the author's personal subscription quota.

Adding This "New User" to the QA Process

After reviewing the report, the author saved the task as a reusable Doubao Work Skill so future releases can be tested the same way. Two next steps are proposed:

Accumulate more such test scenarios as Skills, building a library of agent-executable test cases.

Design a "QA Manager" role that aggregates these reports, integrates with tools like Feishu (Lark), and ensures issues are tracked, fixed, and re-tested.

The author invites discussion on the feasibility of this AI-augmented QA workflow.