Why Traditional Spec‑Driven Tools Fail and How AI‑Powered Context Engineering Can Supercharge Development

Background and Motivation

AI coding assistants have proliferated, but their productivity is limited by large context windows, dynamic enterprise requirements, and tacit knowledge that remains only in developers' heads. This article documents a transition from spec‑driven tools to a multi‑agent, context‑aware AI engineering workflow that reduces marginal cost and compounds knowledge.

Initial Attempts with Spec‑Driven Tools

Speckit Overview

Speckit implements a five‑stage Spec‑Driven Development (SDD) flow: Constitution → Specify → Plan → Tasks → Implement. It enforces nine immutable architectural principles and generates code directly from specifications. The flow works for short‑context tasks (e.g., a single function) but fails for enterprise‑scale projects because it assumes static, well‑defined requirements.

OpenSpec Overview

OpenSpec stores changes as delta operations (ADD, MODIFY, REMOVE, RENAME) and validates them before merging. It aims to keep a lightweight spec history, but delta conflicts require manual resolution and cannot handle constantly changing, multi‑team requirements.

Problems Encountered in Enterprise Context

Dynamic requirements involve many stakeholders and services.

Large context windows overflow, causing loss of important information.

Knowledge about past pitfalls, configuration nuances, and business rules is not encoded for the AI.

Both tools assume a linear, one‑time planning process that does not match iterative development.

Context Engineering and Compound Engineering

Context engineering focuses on managing the token set fed to large language models (LLMs) to maximize signal‑to‑noise ratio. It replaces static prompt engineering with dynamic, layered information retrieval and just‑in‑time loading.

Compound engineering (inspired by Claude Code and Every teams) defines a closed‑loop workflow: Plan → Work → Review → Compound. Each stage produces structured artifacts that become reusable knowledge for future tasks.

Practical AI‑Engineered Workflow

Architecture Overview

The system consists of three layers:

Agent : decides what to do, performs intent recognition, routing, and context management.

Skill : executes concrete steps (e.g., create requirement, generate code, commit).

Command : user entry points such as /req-dev (start or continue a requirement) and /optimize-flow (record experience).

Key Components

Agents: phase‑router, requirement‑manager, design‑manager, implementation‑executor, experience‑depositor.

Skills: req‑create, design‑create, workspace‑setup, code‑commit, experience‑index, etc.

Commands: /req-dev, /optimize-flow.

Directory Structure (Project‑Level Knowledge)

your-project/
├── AGENTS.md                # Project memory entry point
├── .codebuddy/
│   ├── agents/
│   ├── commands/
│   └── skills/
├── context/                # Long‑term knowledge base
│   ├── business/
│   ├── tech/
│   └── experience/
├── requirements/
│   ├── INDEX.md
│   ├── in‑progress/
│   └── completed/
└── workspace/               # Git‑ignored working area

Knowledge Capture and Retrieval

When a problem is solved, the experience-depositor agent writes a markdown file under context/experience/ with sections such as Problem Description, Trigger Conditions, Solution, Verification, and Linked Documents. The experience-index skill is invoked before requirement analysis or design to automatically load relevant files and surface risk alerts.

# context/experience/product-distribution-issue.md
## Problem Description
Product distribution selected the wrong wallet, causing user claim failure.
## Trigger Conditions
- Requirement involves product distribution
- Product type is virtual
## Solution
Virtual products must be sent to virtual wallets, physical products to physical wallets.
## Verification
Check goods_type vs wallet_type matching.
## Linked Documents
- context/tech/Apollo-config-spec.md
- context/tech/services/product-service-summary.md

Just‑In‑Time Context Loading

The agent loads only the most relevant files based on the current stage, keeping the token budget low. Example flow:

1. Intent recognition → phase‑router decides which agent to invoke
2. Agent calls experience‑index with a short description
3. experience‑index returns JSON with context files, risk alerts, service suggestions
4. Agent presents concise reminders to the user

Sub‑Agent Architecture to Avoid Context Overflow

Instead of a monolithic agent, the system spawns lightweight sub‑agents, each with its own context window. State is exchanged via structured JSON files (e.g., requirements/INDEX.md, context/session/), ensuring the main agent never exceeds the LLM’s context limit.

Main Agent → decides tasks → launches SubAgent1, SubAgent2 …
Each SubAgent works independently → writes results to state files → Main Agent aggregates

Model Context Protocol (MCP) Integration

Custom MCP adapters connect the AI workflow to external systems:

TAPD – fetch and update requirement details.

Git (工蜂) – create branches, commit code, open merge requests.

iWiki – retrieve historical technical solutions and business rules.

This eliminates manual clicks and keeps the AI’s knowledge up‑to‑date.

Quantitative Benefits

Cold‑start knowledge loading reduces onboarding time from days to minutes.

Marginal cost drops from a constant ~45 minutes per requirement to under 5 minutes after a few repetitions.

Team productivity improves by more than 70 % in multi‑person scenarios because knowledge is shared via the context repository.

Future Outlook

As LLMs become more capable, auxiliary tools will be absorbed (“model‑eats‑tool” paradigm). The architecture is designed for graceful deprecation: skills and context files can be removed once the model internalizes the behavior. Long‑running agents will evolve from short‑term assistants to autonomous teams that plan, execute, and deliver over days, with humans only setting goals and reviewing final outcomes.

Conclusion

The key insight is to let AI inherit the organization’s tacit knowledge through context engineering, compound engineering, and automated knowledge compounding. This shifts developers from repetitive implementation to high‑level design, supervision, and innovation.

References

Effective Context Engineering for AI Agents – https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

Compound Engineering Plugin – https://github.com/EveryInc/compound-engineering-plugin

OpenSpec – https://github.com/Fission-AI/OpenSpec/tree/main/openspec

Speckit – https://github.com/github/spec-kit

How OpenAI Built Android Sora with Codex – https://openai.com/index/shipping-sora-for-android-with-codex/