Is Your Harness Workflow Actually Improving? A Quantifiable Exam‑Based Evaluation System

Background and Problem Statement

Teams often spend weeks refining Harness CI/CD workflows—adding rules, skills, and prompts—yet after deployment they only have vague feedback such as “it feels more stable” or “it’s smarter”. This reliance on subjective "vibes" means they cannot answer basic questions: Did a rule change improve the workflow? Which side (A or B) is better? How does the workflow regress without a measurable metric?

Core Philosophy: From Tests to Exams

Traditional software testing is binary (pass/fail). Harness workflows are "probabilistic programs" whose output depends on Prompt + Rules + Skills + Model, making a simple boolean assertion insufficient. Therefore the authors propose an exam approach:

Test : verifies correctness (binary).

Exam : evaluates quality across multiple dimensions, provides evidence, and suggests concrete improvements.

Three Non‑Negotiable Principles

Repeatability > Precision : Run a question many times and analyse the distribution of scores rather than a single pass.

Attribution > High Score : Every failure must be traceable to a rule, prompt ambiguity, or model limitation.

Closed‑Loop > One‑Way : Grading must drive the next iteration; a raw number without actionable feedback is useless.

System Overview

The system performs three steps repeatedly: question design , exam execution , and grading . The whole pipeline is fully automated and produces a reproducible, attributable quality report.

4.1 Question Design

Each question consists of four mandatory files (plus optional fixtures): meta.yaml: metadata such as ID, version, category, difficulty. task.md: the user‑facing problem statement and script. rubric.md: hard‑pass criteria, quality items, and typical failure points. env.yaml: environment pre‑checks, required services, and variable placeholders.

Design goals:

Layered question bank (basic → combinatorial → system‑level).

Rubrics must be fully quantifiable; every hard‑pass item must map to an observable artifact in the transcript.

Separate task (what the candidate sees) from rubric (what the judge sees) to avoid “copy‑the‑answer” behaviour.

4.2 Exam Execution

The exam simulates a real multi‑turn interaction between an Examiner (an LLM acting as the user) and the Candidate (the workflow agent). The five steps are:

Environment Setup : env.yaml is validated, required files are placed, and a sandbox is prepared.

Examiner Opens : The examiner reads task.md, adopts the scripted user role, and sends the first message.

Multi‑Turn Interaction : The candidate may ask for clarification, invoke tools (read files, run commands, write code), report progress, or request decisions. The examiner follows the script, asking follow‑up questions or challenging ambiguous actions.

Candidate Finishes : When the candidate believes the task is complete, it emits a final summary; the examiner acknowledges the end.

Full Recording : Every turn, tool call, and shell output is stored in transcript.jsonl, providing an “God‑view” of the execution.

The authors stress that without this interaction the test would only measure the agent’s “self‑play” ability, not its real‑world behaviour.

4.3 Grading Process

Grading is performed by an isolated Judge process that receives only rubric.md and a compressed version of the transcript ( transcript.for-judge.txt). The judge follows a strict workflow:

Verify each hard‑pass criterion against concrete evidence in the transcript; missing evidence = fail.

Assess process quality (was the candidate proactive or reactive? Did it perform dry‑runs?).

Assign three independent scores: Compliance (process adherence), Execution Quality , and Overall .

Quote exact transcript snippets as evidence for every decision.

Generate improvement suggestions, each tagged with one of three categories: [workflow] (rule/skill changes), [eval] (question design issues), [capability] (agent skill gaps).

Example YAML output:

result: pass                # pass | fail
compliance: 4               # 0~5
execution_quality: 4       # 0~5
overall: 4                  # 0~5
summary: "Candidate completed all five nodes, only missed a proactive parameter confirmation."

Improvement suggestions are stored in review.md with concrete evidence lines, e.g.:

reason: 'Proactively confirmed service name / parsed parameters correctly / asked for confirmation before writing.'
evidence:
  - 'Turn 2: "Please choose A (create) or B (skip)" triggered validation skill.'
  - 'Turn 4: appid matched the actual URL value.'
improvements:
  - '[workflow] Turn 1 did not proactively confirm source; add a mandatory gate.'
  - '[eval] Rubric should separate "proactive confirmation" from "post‑question clarification".'
  - '[capability] Multi‑step reasoning often loses key constraints.'

4.4 Execution Engine

The engine is a single Go CLI binary with no external framework dependencies, enabling zero‑install deployment ( go build). Key engineering choices:

Git worktree isolation : each concurrent run gets its own sandbox derived from a shared object store, preventing cross‑run interference.

Symlink swapping : IDE configuration directories are temporarily replaced with the run‑specific snapshot and restored afterwards.

Transient error retries : CLI crashes and network timeouts are retried (2‑3 attempts with short back‑off), while hard timeouts are not retried to avoid token waste.

4.5 Result Reporting

After each batch, the system produces several artefacts (plain‑text and YAML): latest.md – summary table of pass rates, average scores, token usage, and duration per question. latest‑stats.yaml – machine‑readable version of the same data. score‑history.yaml – flat record of every run with the associated workflow_rev (git commit) for pinpointing regressions. batch‑insights.md – aggregated improvement suggestions grouped by the three categories.

Impact on the Team’s Harness Workflow

Applying the framework to the internal Harness pipeline yielded concrete improvements:

Overall pass rate rose from 82.4% (14/17) to 100% (17/17) after targeted fixes.

Key questions that previously scored 1‑2 were upgraded to perfect 5‑point scores.

Systematic exposure of defects in range‑locking, critical gate interception, archiving, commit‑norms, and TDD discipline.

The evaluation infrastructure itself became reusable engineering infrastructure and a methodological white‑paper, ready to be transferred to other workflow contexts.

Conclusion

The authors argue that any probabilistic, “behaviour‑uncertain” workflow—such as Harness—requires a quantifiable, repeatable, attributable, closed‑loop quality measurement system. By turning workflow validation into an exam with evidence‑backed grading, teams can move from intuition‑driven tweaks to data‑driven continuous improvement.