Dissecting OpenWorker’s Desktop Agent Architecture – the Open‑Source “WorkBuddy”

Overview of OpenWorker’s Architecture

OpenWorker’s official README shows a three‑layer diagram: Desktop Application → Local Agent Service → Files/Tools/Model. The author refines this into four responsibility groups:

Interaction Layer : React + Tauri UI that displays sessions, progress, deliverables, approval cards, and settings.

Agent Runtime Layer : Python service that runs the task loop, invokes models, assembles skills, orchestrates tools, and manages the workspace.

Control Layer : Risk classification, permission decisions, Inbox, and Audit Store that constrain actions.

Resource & Execution Layer : Files, terminal, connectors, MCP, and model providers that form the real work environment.

These four groups are illustrated in the following six running lines that cooperate during a task:

Task line – how the goal is split into steps and progress is updated.

Model line – how different models are plugged in and decoupled from tools.

Tool line – unified calls to files, search, commands, and external systems.

Permission line – classification of actions and whether they need approval.

Status line – how front‑end sessions, back‑end tasks, pending approvals, and audit logs are linked.

Delivery line – how results become files, messages, or calendar changes instead of just a text reply.

Key Source Modules

coworker/agents/cowork.py

– defines the Cowork Agent’s goals, capabilities, and delivery contract. coworker/agent.py – assembles model, tools, skills, workspace, and executor. coworker/engine.py – handles a dialogue round: model call, tool authorization, execution, and result back‑fill. coworker/risk.py and coworker/permissions.py – implement risk classification and permission decisions. coworker/inbox.py and coworker/audit.py – provide the Inbox queue and audit store. coworker/skills/base.py and coworker/tools/shell.py – implement skill loading and local command execution.

Step‑by‑Step Execution of a Sample Task

1. Desktop passes the goal to the local service

The desktop creates a session and workspace context, then forwards the user request to the local Agent service. Model keys, connector tokens, and session state are stored locally.

2. Agent creates visible progress

The Cowork Agent has four core capabilities:

COWORK_CAPABILITIES = ["files", "search", "shell", "todo"]

When a tool is needed, the system first writes a todo_write entry with an in_progress status. The desktop Progress panel reads this state, giving the user immediate feedback.

3. Model proposes actions; tool layer executes them

The model never manipulates the computer directly. It emits tool‑call specifications, which the Engine forwards to file, search, terminal, or connector implementations. OpenWorker ships with over twenty connectors and also supports MCP.

If the model proposes multiple calls, the Engine authorizes each one sequentially. Low‑risk reads may run in parallel, while writes and commands are executed serially to avoid conflicts and keep execution records clear.

4. Risk classification before execution

Actions are classified into four risk types:

READ : read files, search, query – no side effects, executed directly.

WRITE_LOCAL : write files or modify content – requires write‑mode checks and allowed directories.

EXEC : run commands – checked against allowed command lists and approval.

EXTERNAL : send messages, modify calendars, or call external APIs – needs explicit authorization.

The Permission Engine decides based on the current mode (Discuss, Plan, Interactive, Custom, Auto). For example, Interactive mode allows reads automatically, but writes, commands, and external actions usually need confirmation.

5. Front‑end approval and back‑end Inbox sharing

When a human is present, approval cards appear in the current session. Background tasks that encounter the same action are paused in the Inbox until a user approves, rejects, or provides additional information. This distinction makes “generate weekly report” (a draft) different from “send weekly report” (an external effect).

6. Result stored as Artifacts and audit records

The Agent issues an artifact: link; the desktop Artifacts panel can display, open, and refresh the deliverable. Simultaneously, the Audit Store records tool usage, execution stage, status, approval outcome, parameter summary, and resource references. Thus each task yields three states: Progress, Artifacts, and Audit.

Model Swappability and Stable Runtime Contracts

OpenWorker can use OpenAI, Anthropic, Gemini, GLM, DeepSeek, Kimi, Qwen, MiniMax, or local models via Ollama. Model choice affects tool‑call stability, context length, instruction following, and cost, but the runtime contracts (tool parameters, risk classification, workspace, progress, audit, and artifact links) remain unchanged.

Skills – Method Layer Separate from Permissions

Skills are described in SKILL.md with YAML metadata, Markdown instructions, and optional resources. They are loaded lazily via load_skill only when needed, avoiding flooding the model context.

Example: a “generate architecture review report” skill may specify:

Read ADRs, interface docs, and incident logs.

Compare goals, constraints, alternatives, and rollback paths.

Output a risk list and pending questions.

Produce a fixed‑format review document.

The skill defines the method; the permission engine still decides whether the resulting actions (e.g., sending to Slack) are allowed.

Execution Boundary – LocalExecutor

OpenWorker’s command tools sit on an Executor abstraction; the current implementation is LocalExecutor, which keeps a persistent shell so that cd, environment variables, and virtual environments survive across commands—convenient for development and data processing.

The trade‑off is that commands run in the local environment without strong isolation. Future interfaces for ContainerExecutor and VMExecutor are planned but not yet functional.

Comparison with Tencent WorkBuddy

Product form : WorkBuddy is a Tencent Cloud product within an enterprise ecosystem; OpenWorker is an MIT‑licensed open‑source desktop app.

Model & account : WorkBuddy uses Tencent‑provided models and services; OpenWorker lets users bring their own keys or run models locally via Ollama.

Tool integration : WorkBuddy relies on Tencent’s platform capabilities; OpenWorker connects to local files, terminals, connectors, and MCP.

Runtime base : WorkBuddy emphasizes an Agent Runtime with a security sandbox; OpenWorker runs a local Python service with the current LocalExecutor.

Observation focus : WorkBuddy highlights enterprise workbench and managed runtime; OpenWorker exposes the desktop Agent’s modules, protocols, and source implementation.

Both aim to answer “how AI can finish a task,” but they choose different infrastructure paths.

Four Representative Tasks to Exercise the Architecture

Work : Turn Monday meeting notes and project data into an HTML/Markdown brief.

Life : Convert scattered travel notes, budget, and dates into an itinerary; calendar changes trigger the permission flow.

Architecture : Gather ADRs, interface docs, and incident records to produce an architecture review draft.

R&D : Read repository state, issues, test output, and change logs; optionally run approved checks and generate a release risk list.

Each task follows the same pipeline: goal → permission‑checked tool calls → artifact generation → audit logging.

Final Thoughts

OpenWorker is still in open beta. Its value lies not only in being a downloadable desktop app but also in providing a readable, runnable codebase that demonstrates how a desktop Agent can be built:

Model proposes the next step.

Engine organizes the loop.

Tools connect to the real environment.

Permission Engine enforces risk policies.

Inbox pauses for human decisions.

Audit Store records the process.

Artifacts store the final product.

Limitations include the lack of strong isolation for the local executor, approvals not replacing result verification, and the possibility of models or connectors sending data externally. These boundaries are explicit in the code, making it clear where future improvements are needed.

The author concludes with four simple questions that the architecture is designed to answer: Is progress visible? Are actions controlled? Is the process traceable? Is the final product openable?

References

OpenWorker official site: https://openworker.com/

OpenWorker GitHub repository: https://github.com/andrewyng/openworker

aisuite repository: https://github.com/andrewyng/aisuite

OpenWorker README: https://github.com/andrewyng/openworker/blob/01b6f83b3927e02912dda84bb392942c13ca70d1/README.md

Cowork Agent source: https://github.com/andrewyng/openworker/blob/01b6f83b3927e02912dda84bb392942c13ca70d1/coworker/agents/cowork.py

Agent assembly: https://github.com/andrewyng/openworker/blob/01b6f83b3927e02912dda84bb392942c13ca70d1/coworker/agent.py

Permission engine & risk classification: https://github.com/andrewyng/openworker/blob/01b6f83b3927e02912dda84bb392942c13ca70d1/coworker/permissions.py and https://github.com/andrewyng/openworker/blob/01b6f83b3927e02912dda84bb392942c13ca70d1/coworker/risk.py

Engine, Inbox, Audit Store: https://github.com/andrewyng/openworker/blob/01b6f83b3927e02912dda84bb392942c13ca70d1/coworker/engine.py, https://github.com/andrewyng/openworker/blob/01b6f83b3927e02912dda84bb392942c13ca70d1/coworker/inbox.py, https://github.com/andrewyng/openworker/blob/01b6f83b3927e02912dda84bb392942c13ca70d1/coworker/audit.py

Skill loader & Shell executor: https://github.com/andrewyng/openworker/blob/01b6f83b3927e02912dda84bb392942c13ca70d1/coworker/skills/base.py, https://github.com/andrewyng/openworker/blob/01b6f83b3927e02912dda84bb392942c13ca70d1/coworker/tools/shell.py

Tencent WorkBuddy official intro: https://www.tencent.com/zh-cn/articles/2202341.html

Tencent WorkBuddy documentation: https://intl.cloud.tencent.com/zh/document/product/1300/80640