Design Retrospective: Splitting Tools and Delegating Judgment to AI

01 | Nine Pitfalls

During development of an AI‑focused CLI suite we encountered nine distinct problems that highlighted why judgment must be separated from execution.

Pitfall 1: Command tree mirrors backend API paths. Directly copying API routes into CLI commands made both AI and humans unable to understand the commands, and any backend rename broke the CLI instantly.

Pitfall 2: Internal endpoints leaked to AI‑visible commands. Exposing the full backend list let the AI list private endpoints it could not use, creating accidental calls.

Pitfall 3: Raw role interfaces passed to AI for JSON construction. Ambiguous naming (e.g., an "editor" role used for two different concepts) caused the AI to write data to the wrong role, resulting in data loss.

Pitfall 4: Command renaming and mismatched naming broke AI workflows. Inconsistent names across package, binary, and documentation caused AI scripts to fail overnight.

Pitfall 5: Cross‑platform process execution failures. On Windows the CLI spawned npm without the .cmd extension, leading to "command not found" errors and exposing fragile environment‑variable assumptions.

Pitfall 6: Login flow hard‑coded to open a browser. In headless automation the CLI hung because it waited for a user to click a browser‑based authorization.

Pitfall 7: CLI exposed a pagination‑enabled organization search. Unrestricted paging allowed the AI to dump the entire company hierarchy, a serious security risk.

Pitfall 8: Irreversible actions executed without confirmation. The AI created resources and bound Git repositories without checking the workspace state or asking for human approval.

Pitfall 9: Implicit parameter constraints caused missing or wrong fields. Nested required fields were hidden in documentation, leading the AI to submit incomplete requests.

02 | Core Insight: Separate Judgment and Execution

After reviewing the pitfalls we concluded that a tool for AI must place deterministic execution, constraints, and confirmations in the tool layer, while leaving ambiguous judgment to an AI that can converse and reason.

Key rules emerged:

CLI commands should be handcrafted, task‑oriented, and semantically clear—not a one‑to‑one mapping of backend endpoints.

Each backend addition does not automatically become a new CLI command.

Execution layer handles "where to write" and "how to write" with certainty.

Judgment layer decides "which type of entity" or "which group" the user actually intends, based on context and clarification.

03 | Skill Packaging with the CLI

The judgment logic is stored as Markdown files (SKILL.md and a set of reference/*.md) that are bundled with the CLI distribution. During installation the files are copied into the IDE assistants' skill directories, allowing the AI to read them at runtime.

Benefits of this approach:

SOP updates require no new binary release; a single Markdown change takes effect immediately.

Consistent behavior across multiple IDE assistants.

Safety: the AI can only invoke commands described in the skill files, preventing it from inventing unsupported operations.

cli-repo/skill/<skill>/
├── SKILL.md               # entry: mental model, generic judgment, high‑risk rules
└── references/
    ├── create.md          # creation: spec confirmation, Git metadata checks
    ├── develop.md         # development / configuration / deployment
    ├── iterate.md         # iteration: diagnostics, sync, retrospection
    ├── use.md             # conversation, session, usage tasks
    ├── assets.md          # skill / knowledge / MCP resources
    └── tools.md           # Git, model queries, auxiliary utilities

Four practical guidelines for maintaining this structure:

Write mutable judgment rules in Markdown, keep stable commands in code.

Package the skill directory with the CLI so the AI environment receives it automatically.

Let the CLI perform deterministic data retrieval; let the AI read the skill files to decide next steps.

For high‑risk actions, embed explicit "explain impact then confirm" steps in the skill files.

04 | Real‑World Case 1: Misrouting

The platform distinguishes two role groups: "members" and "collaborators." An early CLI command config roles exposed the raw role interface, causing the AI to add a user to the wrong group because the term "developer" was overloaded.

Fix:

Remove the public config roles command; keep the raw interface internal.

Introduce two explicit commands: config consumers (add user) and config collaborators (add collaborator).

Document in the skill that ambiguous additions default to the consumer path.

When multiple groups are possible, make the CLI exit with a non‑zero code and list the options, forcing the AI to defer to a human.

Core idea: downgrade low‑level interfaces to internal use and let the skill layer handle semantic routing.

05 | Real‑World Case 2: Data‑Leakage Entry

The organization search API supported pagination and optional keywords, which the CLI initially exposed unchanged. In a security review we realized the AI could iterate through pages and dump the entire org chart.

Fix:

Make keyword a required parameter; reject empty values.

Hard‑code page=1, page_size=5 to limit results.

Strip sensitive fields such as parent_org_id from the output.

Require the flow search → select → consumers add; restrict delete operations to authorized scopes.

Lesson: OpenAPI can expose full capabilities, but a CLI aimed at AI must proactively tighten risky surfaces.

Summary

1. Separate judgment from execution. Deterministic actions, constraints, and confirmations belong to the tool; ambiguous decisions belong to a conversational AI.

2. CLI commands must be task‑oriented, not a direct translation of backend paths.

3. Decouple fast‑changing judgment logic from slow‑changing code. Store judgment in Markdown so updates require no new binary.

4. Never expose internal interfaces directly; always gate them with explicit, semantic commands.

5. Package skills with the CLI to ensure consistent, auditable AI behavior across environments.

6. High‑risk or irreversible actions must include an explicit human‑confirmation step.