How to Capture Expert Knowledge When AI Coding Joins Your Team

Why AI Coding Teams Need More Than Prompt Libraries

After two months of talking to several product‑research teams, the author observes that the real pain point is not "whether to use AI" but the difficulty of replicating the implicit engineering experience of experts when AI tools like Claude Code, Codex, or Cursor Agent are introduced.

People often blame differences on prompts, yet a deeper analysis of real tasks shows that the gap lies in what the expert does before and after prompting: gathering missing context, identifying risks that the model should not decide, recording evidence, and feeding failures back into the process.

The author introduces a simple term – process (工序) – to describe a narrow, repeatable sequence of actions that an Agent can execute, similar to a worker on an assembly line.

Key Insight

AI programming in a team should focus on preserving reusable, inspectable, and trim‑able Agent processes, not just a collection of pretty prompts.

Team‑wide AI performance gaps are usually caused by hidden engineering experience that is not captured.

“Distilling expert experience into Agents” means breaking stable expert actions into runnable processes.

Experts add value before the prompt (supplying context, setting boundaries) and after the model output (diff checking, logging, writing back rules).

An Agent process must specify when it starts, what materials it reads, what checks it performs, when it stops, and who confirms the result.

Start small: pick a high‑rework area, run a shadow pilot for 30 days, and see if it surfaces real omissions early.

Measure value beyond saved hours – look for tighter reviews, earlier detection of rework, clearer hand‑offs, and knowledge that flows to the next iteration.

Prompt Libraries Are Only the Entry Point

Many teams start by collecting prompts such as "Requirement analysis Prompt", "Architecture Prompt", "Test case Prompt", and "Code review Prompt". While useful, prompts only record what is said to the model. The richer expert knowledge lies in the actions taken before the prompt (context gathering, boundary setting) and after the model answer (verification, evidence collection).

When an expert receives a vague requirement, they first identify which business flow is affected, which legacy constraints must stay intact, which data definitions cannot change, and where human sign‑off is needed. They then supplement the AI with context (APIs, old system constraints, gray‑release strategy, test commands, coding style, past pitfalls). After the AI generates code, the expert reviews diffs, runs tests, checks logs, simulates real paths, and writes any failure reasons back into rules, scripts, skills, or checklists.

If this knowledge stays only in one person's head, newcomers cannot learn it, ordinary members cannot reuse it, and leaders cannot assess the reliability of AI output. Consequently, only a few individuals become stronger with AI while the rest become more cautious.

Start with Work Traces

The author recommends a lightweight retrospective: pick a few real tasks and trace back through conversation logs, diffs, document changes, and review comments.

When does the expert add missing context?

When does the AI stop and why?

What does the AI change and what does the human change?

What follow‑up questions does the reviewer ask, and how is the issue finally closed?

This exercise reveals recurring actions (e.g., checking roles, abnormal flows, permissions, state, acceptance criteria, instrumentation, and operational configuration) that can be codified into a process.

What a Single Process Should Declare

Using a "Requirement Integrity Check" as an example, the process does not decide business priority or directly modify the scope; it only surfaces omissions, contradictions, and pending confirmations before a review.

When the PRD is ready for review, the Agent reads the PRD, prototype, historical requirements, and domain rules, checking only for omissions, contradictions, and items needing confirmation; if priority decisions, scope trade‑offs, or insufficient materials are encountered, it stops for confirmation by the product and technical owners.

The full specification can be written as:

Name: Requirement Integrity Check
Entry condition: PRD or prototype is drafted and ready for review
Input materials: PRD, prototype, historical requirements, domain rules, instrumentation specs, related customer‑service feedback
Agent actions: Verify roles, main flow, exception flow, permissions, state, acceptance criteria, instrumentation, and operational configuration
Stop condition: List missing materials; stop on business priority or scope trade‑offs without making decisions
Output: List of omissions, contradictions, pending questions, and suggested supplemental materials
Human confirmation: Product and technical owners decide what to change or accept risk
Write‑back: Frequently missing items are fed back into the PRD template, review checklist, or Skill

The value of this simple process lies in putting the entry point, input materials, stop points, confirmation owners, and write‑back path on a single sheet, so the Agent does not have to guess business logic. When materials are insufficient or a scope decision is needed, the process knows exactly where to pause and who to involve.

Prioritize Upstream Processes

After AI coding becomes popular, teams naturally try to let AI write more code because code is visible and feedback is fast. However, upstream errors amplify downstream rework. Unclear requirement boundaries cause rapid but costly rework; missed impact scopes later surface as permission, data, billing, or compliance issues; vague acceptance criteria force testers to rely on personal experience.

Therefore, the author suggests selecting the first Agent processes based on four criteria: they occur repeatedly, inputs can be collected, outputs are reviewable, and failure radius is controllable. Typical starter scenarios include:

Requirement integrity check

Architecture impact scope pre‑screen

Test point completion

Release risk check

Post‑incident or rework rule write‑back

These tasks are not flashy but align closely with real team pain points, have clear boundaries, produce inspectable results, and keep failure costs manageable.

Roles as Fixed Checkpoints, Not Virtual Jobs

Names like "Product Manager Agent", "Architect Agent", or "Test Agent" sound attractive but can lead teams to expect the Agent to understand goals, organize flow, and make decisions autonomously. A more controllable approach treats role‑based Agents as narrow inspection viewpoints within a process, e.g., a product view checks requirement gaps, an architecture view checks system constraints, a test view checks boundary coverage, and a security view checks permissions and data inputs.

These fixed checkpoints are easier to validate than a full virtual team and allow concrete metrics such as false‑positive rate, missed‑issue rate, and review‑time reduction to be recorded.

Measuring Value Through Reduced Rework

Many AI pilots report saved hours, but that metric alone is misleading. The author recommends looking for concrete changes in the workflow:

Before reviews, problems that previously surfaced during development or testing are now highlighted earlier.

Review meetings spend less time re‑explaining background and more time on critical trade‑offs.

Hand‑offs become clearer: the next person instantly knows the inputs, checked items, open questions, and owners.

Failures that are not captured in templates, Skills, scripts, CI gates, or review checklists are likely to recur.

A stable Agent process makes key issues appear earlier, concentrates human judgment, and makes experience easier to retain.

30‑Day Shadow Pilot

To try the approach, avoid building a full platform at first. Choose a high‑rework area that has repeated material gaps or missing dependencies over the past three months. Examine three to five real tasks, note what the expert asks, what context they add, which AI segment they run, how they modify the output, and what reviewers later question.

Write a one‑page process, run it in shadow mode for 30 days, and compare the Agent‑found issues with the human‑found ones. If the shadow run is stable, gradually expose the process to low‑risk low‑volume flows.

Key rhythm: read more, write less, keep evidence clear, and have human confirmation. At month‑end, prune the process – delete never‑hit checks, downgrade noisy rules, and feed high‑value recurring findings back into templates or Skills.

Boundaries and Growth

Not every expert habit should be codified; some are project‑specific or personal preferences. Only capture actions that repeat, are verifiable, explainable, and reusable for the team.

Agents can surface suggestions, but final decisions still belong to humans (product owners, architects, security leads). Newcomers should follow the process while gradually learning the underlying reasoning, otherwise the team ends up with people who can run checklists but lack system understanding.

Final Thought

AI tools will keep evolving, but the real differentiator for teams is the ability to turn the hidden actions of a few experts – clarifying problems, supplementing context, assessing risk, verifying results, and writing back experience – into machine‑readable, runnable, and inspectable processes. When those processes are clear about input, procedure, evidence, and hand‑off, the team’s baseline improves steadily.

References

Addy Osmani – Loop Engineering [1]

Omar / DAIR.AI – From Prompting Agents to Loop Engineering [2]

Claire Vo / ChatPRD – Designing AI Agent Loops in Claude Code and Codex [3]

Andrew Ng – Three Key Loops for Building Great Software [4]

Anthropic Claude Code Docs – Commands, Automate work with routines, Dynamic workflows [5‑7]

OpenAI Codex Docs – Follow a goal, Agent Skills [8‑9]