Building Self-Evolving AI Skills: A Harness Engineering Case Study with SkillForge

AI Agent is moving from demo to production. The real question is no longer "Can AI do the work?" but "How can AI do the work reliably and continuously?"

Recent model releases are dense with keywords like Agentic, Coding, and support for tools such as OpenClaw and Hermes Agent, signaling that AI agents are becoming an industry consensus.

Harness Engineering resonates because it treats AI not as a one‑off call but as a system that can work continuously and reliably on complex tasks.

If you want a real‑world Harness Engineering setup to run smoothly, expect at least a year of iteration. Reliability comes from repeated adjustments in real tasks, errors, and feedback.

1. Understanding Harness Engineering

Harness Engineering, or “harness engineering,” is not about better prompts or more complex workflows; it is about building an environment where an AI agent can work long‑term with stability, reliability, and iterability.

Its most common scenario is code development, but any sufficiently long and complex task that requires rules, context, feedback, and continuous correction can use this approach.

A functional Harness includes:

Documentation: tells the AI the project, goals, and history.

Standards: defines what counts as good output.

Constraints: defines prohibited actions and boundaries.

Process: guides the AI on how to advance the task.

Feedback: lets the AI receive improvement signals from results.

Iteration: allows the AI to solidify effective experience for future runs.

These elements together form a usable Harness.

In other words, Harness Engineering solves the problem of "one‑off AI output" by enabling AI to improve continuously within a system.

2. SkillForge: A Small‑Scale Harness Practice

To test the concept, I built a project called SkillForge with the simple idea:

Give the AI a book, have it extract the core methodology, generate an executable Skill, publish it to GitHub, and create a showcase webpage.

The goal is 24/7 unattended operation, with occasional checks.

3. Step 1 – Turning a Vague Idea into Requirements

I started with a rough notion: can AI turn classic book methods into Skills? I refined this by discussing with the AI and producing a requirements.md that answered basic questions about problem scope, inputs, outputs, success criteria, automation points, and manual checks.

4. Step 2 – Splitting into Composable Skills

SkillForge’s goal requires a chain of actions: understand the book, extract methodology, turn it into a Skill, generate examples, publish to GitHub, and create web content. I therefore created six separate Skills:

Main Skill – overall judgment and scheduling.

Reading Skill – parses book structure and core content.

Extraction Skill – pulls out methodology and executable framework.

Publishing Skill – organizes results and pushes to GitHub.

Example Skill – generates demonstration cases.

Web Content Skill – creates user‑facing description.

Skill design is engineering work, not just copy‑writing; it requires validation, parameter tuning, and iterative refinement.

5. Step 3 – Writing an Anchoring Document

I authored a What is SkillForge document that answers what SkillForge is, what it is not, its style, scope, and the user experience it serves. This prevents the AI from “over‑engineering” by keeping it focused on the core goal.

6. Step 4 – Enabling Skill Evolution During Runtime

Each Skill stores Markdown artifacts (reference, spec, execution logs, improvement suggestions). When the AI encounters unclear rules or unstable outputs, it updates these files, and the next run automatically incorporates the new knowledge.

This creates a self‑learning loop where the Skill becomes a living system rather than a static file.

The loop follows four steps: Generator creates output, Evaluator checks against standards, Re‑generator revises based on feedback, and the effective experience is sunk back into the Skill or documentation.

Anthropic’s "Building Effective Agents" article illustrates this pattern, showing how multiple evaluation‑feedback cycles dramatically improve quality.

7. Three Key Points for Successful Harness Engineering

1. Documentation Is the Agent’s Infrastructure

Effective alignment starts with documentation. A robust Harness includes Architecture, Execution Plan, Technical Debt, Backlog, Spec, and Progress documents, forming a project map that the AI consults before acting.

2. Session Management Determines Long‑Task Quality

As sessions grow, context becomes noisy. I let the AI generate a progress.md when the conversation gets tangled, summarizing completed work, current status, key decisions, open issues, and the next starting point. New sessions resume from this hand‑off, preserving continuity.

3. Generator + Evaluator Drives Self‑Iteration

The closed‑loop of generation, evaluation, re‑generation, and experience sinking is essential; without it, AI stalls at the first draft.

8. Skills Are Core, Not Sexy

High‑quality, optimizable Skills dramatically boost agent performance. Skills should be reusable, composable, and capable of continuous improvement.

9. Skills Are Not the End Goal

The purpose of writing Skills is to solve real problems, not to showcase AI capability. Real‑world projects—like digitizing handwritten receipts for an optical store—highlight the tangible value of well‑engineered Skills.

These projects involve varied document formats, writing habits, accuracy requirements, error‑free ordering, traceable exceptions, and minimal staff burden—issues that are concrete, not glamorous, yet highly valuable when solved.

10. Final Thoughts

By 2026 AI models will keep improving, and Agent engineering will accelerate. The most worthwhile investment is not chasing concepts but building reliable Harnesses and reusable Skills that address specific, high‑impact problems.