Skip to content
AI Agent Checklist That Works: 5 Dimensions, Binary Assertions, Real Consequences

AI Agent Checklist That Works: 5 Dimensions, Binary Assertions, Real Consequences

This article presents a practical, takeaway self-checklist for AI agent teams, structured around five dimensions—evaluation, context, multi-agent, long tasks, and self-evolution—each with binary-decidable assertions, validity checks, and concrete consequences for failure, replacing vague adjectives with verifiable criteria and a three-state verdict system to prevent false passes.

James' Growth Diary
James' Growth Diary
James' Growth Diary

The article addresses a practical problem: how to convert a conceptual evaluation loop (evaluation, context, multi-agent, long tasks, self-evolution) into a usable team checklist. It identifies three failure modes observed in internal reviews: (1) unjudgeable items using adjectives like "sufficient" default to "should be fine"; (2) checklists without consequences degrade into ritual — 40 items all pass, incident occurs two weeks later; (3) checklists verify mechanism existence, not effectiveness — a timeout fallback existed in code but had been bypassed in a refactor three months earlier.

Terminology

Agent Platform : platform for developers to create agents, not a single agent

Agent : specific agent produced on the platform

Self-Checklist : table the team confirms item by item before delivery/launch

Ruler : a dimension's check collection; five rulers: evaluation, context, multi-agent, long tasks, self-evolution

Assertion : a check item decidable yes/no on the spot

Existence Check : asks "does this mechanism exist?"

Validity Check : asks "does this mechanism actually work?"

Three-State Verdict : Pass / Fail / Untestable — "Untestable" cannot count as pass

Consequence : specific thing that must happen when verdict is Fail

Rewrite : after an incident, add a new assertion to catch the issue next time

Analogy: treat the self-checklist like a cockpit pre-flight checklist — two people go through it together, each item only Yes/No, one Fail means no takeoff, divided by phase, paper-based/offline, revised after every accident.

Three Checklist Failure Scenes

Silence halfway through : "Is context management sufficient?" — nobody can define "sufficient." An unjudgeable item equals a non-existent item.

Forty items all pass, incident two weeks later : item count exceeds attention span; full pass carries zero cost. A checklist without consequences degrades into ritual.

Postmortem reveals the item was ticked : checklist asked "Does critical path have timeout fallback?" Team ticked "Yes" — code had it, but fallback was bypassed in a refactor three months earlier. Checklist verifies "existence"; accident verifies "effectiveness" — a credibility gap lies between.

Why "Listing a Table" Isn't Enough: Three Fractures

Mistaking "Having a Checklist" for "Using a Checklist" : a checklist with no bound action is skipped or perfunctorily completed. It must have a definite position — what event must it precede, and that event cannot happen until the checklist passes.

Items Undecidable : adjectives ("sufficient," "reasonable," "timely," "robust") cannot serve as criteria. Criteria must be pointable: does this file exist, is this number greater than that threshold, does this run's result match the last run's.

Only Verifying Existence, Not Effectiveness : "Does a timeout fallback exist?" vs "Will the timeout fallback actually trigger?" Existence checks are cheap but weak; validity checks are expensive but real. A checklist with only existence items gives false sense of security.

Four approaches compared:

One 80-item mega-table: steals coverage, rejects executability (nobody reads it, degrades into ritual)

Five rulers, 3-5 items each: steals executability, rejects incomplete coverage (must accept)

Only verify existence: steals cheap/fast, rejects false sense of security

Only verify validity: steals real, rejects cost too high to run

Overall Approach: Decidable × Validity × Consequence

Every assertion must satisfy three things simultaneously: binary decidable, two layers (existence + validity), Fail has consequence.

A qualified assertion = a question decidable on the spot + a "how to prove it really works" follow-up + a "what happens if it fails" answer.

Three-Layer Structure (chained)

Decidable Layer : Can this item be pointed at and answered Yes/No? Answerable on the spot, no discussion needed. If fails → item is waste.

Validity Layer : Even if "exists," does it actually work? Requires manufacturing the condition, higher cost. If skipped → item deceives.

Consequence Layer : If Fail, what gets stopped? Requires team pre-agreement. If missing → item gets perfunctorily passed.

Cockpit analogy: "Flap lever down" (decidable); "Flaps actually in position" (validity — lever down but stuck, light illuminates); "Not in position → no runway entry" (consequence). Missing any one, the checklist is just paper.

Key Design 1: Assertions Must Be On-The-Spot Decidable

4.1 Replace Adjectives with Assertions

Rewrite pattern: swap "good/bad" for "exists/doesn't" or "exceeds/doesn't exceed." Real rewrites:

"Is context management sufficient?" → "Is the single-turn input budget upper limit written in a readable place?"

"Are long tasks stable enough?" → "Does a single task have an explicit timeout upper limit, and does timeout terminate or continue?"

"Is multi-agent collaboration clear?" → "In one dispatch, does each sub-task's result have a unique identifier?"

"Is evaluation useful?" → "Can the score difference before/after the last change on the same test batch be retrieved?"

"Is evolution safe?" → "Does the evolution-produced change have a mandatory human-confirmation gate?"

Only rewrite criterion: after reading the item, can someone in the room immediately answer Yes or No?

4.2 Three States: Pass / Fail / Untestable

Binary isn't enough; many items are genuinely untestable (environment not ready, data unavailable, dependency missing). Most common error: recording "Untestable" as "Pass." Rule fixed: "Untestable" and "Fail" recorded separately in statistics, but treated equally for gating — neither can serve as a pass basis. To turn it into Pass, you must add the measurement capability.

// TypeScript illustration
type Verdict = "PASS" | "FAIL" | "UNKNOWN";

function gate(rows: Array<{ id: string; verdict: Verdict }>) {
  const blocking = rows.filter((r) => r.verdict !== "PASS");
  return {
    pass: blocking.length === 0,
    blocking, // FAIL and UNKNOWN both block
    unknown: rows.filter((r) => r.verdict === "UNKNOWN"),
  };
}
# Python illustration
def gate(rows):
    blocking = [r for r in rows if r["verdict"] != "PASS"]
    return {
        "pass": not blocking,
        "blocking": blocking,
        "unknown": [r for r in rows if r["verdict"] == "UNKNOWN"],
    }

Separately listing "Untestable" counts becomes a to-do list — which measurement capabilities are still missing, visible at a glance.

4.3 Every Item Must State "What Happens on Fail"

Only one way to write it: point to a specific thing that will stop.

Single-turn budget upper limit readable → Cannot enter canary

Long task has timeout upper limit → Cannot open to external users

Sub-task result has unique ownership → Cannot enable multi-user collaboration entry

Evolution change has human-confirmation gate → Evolution result must not be merged back

An item that Fails but has no consequence is not a check item, it's a wish item.

Key Design 2: Five Rulers, Check Item by Item

Each ruler provides: Existence Assertion (on-the-spot decidable), Validity Follow-up (how to prove real effectiveness), Fail Consequence (what stops).

5.1 Ruler 1: Evaluation — Can You Prove "Change Made It Better"?

Existence: Is there a fixed test batch runnable before and after changes? Validity: Can the score difference before/after the last change be retrieved on the spot? Consequence: Cannot claim "change made it better"

Existence: Are scoring dimensions fixed, not changed each run? Validity: If another person runs the same batch, are results consistent? Consequence: Cannot use this batch for decisions

Existence: Is there an "Untestable" state? Validity: Are "Untestable" items being counted as "Pass"? Consequence: This evaluation conclusion void

5.2 Ruler 2: Context Engineering — What Exactly Should Enter This Turn?

Existence: Is the single-turn input budget upper limit written in a readable place? Validity: Create an over-long session — does truncation actually happen? Consequence: Cannot enter canary

Existence: Is the prompt assembled in three replaceable segments? Validity: Change one segment — do the other two remain untouched? Consequence: Cannot adjust prompt

Existence: Is the tool list delivered on-demand, not full? Validity: Remove one tool — does the model still try to call it? Consequence: Cannot add new tools

5.3 Ruler 3: Multi-Agent — Dispatch & Ownership

Existence: In one dispatch, does each sub-task have a unique identifier? Validity: When a sub-task fails, can it be retried alone? Consequence: Cannot enable task dispatch

Existence: Is sub-task output ownership explicit? Validity: Two sub-tasks writing the same file — will they overwrite each other? Consequence: Cannot allow concurrent multi-sub-task writes

Existence: Do collaboration events have stable semantics, not free text? Validity: Switch host (page / IM) — does event parsing stay consistent? Consequence: Cannot integrate new host

5.4 Ruler 4: Long Tasks — What Keeps a 10+ Hour Run from Crashing?

Existence: Does a single task have an explicit timeout upper limit? Validity: Actually run a timeout — does it terminate or continue? Consequence: Cannot open to external users

Existence: Is there a heartbeat, and will a missed heartbeat be detected? Validity: Suspend process for 10 minutes — how long until system notices? Consequence: Cannot promise task reliability

Existence: On partial failure, are completed parts preserved? Validity: Restart after mid-run failure — does it resume or start over? Consequence: Cannot use for tasks with artifacts

Existence: Is artifact ownership recorded on the session? Validity: Task spans multiple messages — can artifacts still be traced? Consequence: Cannot show "this round's artifacts"

5.5 Ruler 5: Self-Evolution — Let the Agent Improve the Agent

Existence: Does the evolution-produced change have a mandatory human-confirmation gate? Validity: If unconfirmed, does the change auto-apply? Consequence: Evolution result must not be merged back

Existence: Can the same evolution run be executed repeatedly? Validity: Click twice — is it recorded only once? Consequence: Cannot enable manual trigger

Existence: Is the evolution's data traceable to specific sessions? Validity: Pick any conclusion — can you trace back to which task it came from? Consequence: Conclusion cannot serve as change basis

5.6 How to Run This Checklist

Three agreements matter more than the checklist itself:

Two people together. One reads, one judges — commitment differs by an order of magnitude.

No more than 25 items per run. Five rulers, pick the five most critical each. Human attention has a ceiling; beyond it, all "Yes" is inevitable.

One run, 30 minutes max. Overtime stops; unfinished parts get a separate slot.

Back to the cockpit: a checklist is never better for being longer — it's the maximum number of items that can be seriously completed in those pre-takeoff minutes.

Acceptance: How Three Scenarios Play Out

Scenario 1 (Easy): New Team First Time Using This Checklist

Decision A: Give full dozens of items, let them pick priorities. Decision B: First run only 25 items — five rulers × five each, get it running smoothly then add. Choose B. Checklist credibility is accumulated: first let the team experience "a real item got a Fail, and that thing actually stopped." Hand a big table upfront, first run stalls, the table dies on run one.

Scenario 2 (Medium): Review Meeting, 40 Items All Pass

Check three things first: Are items adjectives? (If yes, fundamentally undecidable); Is there an "Untestable" state? (If not, all undecidable items recorded as "Pass"); Does Fail have a consequence? (If not, ticking "Pass" is zero-cost). Decision A: Lengthen the table. Decision B: Introduce "Untestable" with separate counting, and bind a consequence to every item. Choose B. Seriousness doesn't grow with item count — it grows with "Fail has consequence."

Scenario 3 (Hard): Postmortem Reveals the Item Was Ticked

Exposes the checklist's layer was wrong: item verified "existence"; accident verified "effectiveness." Decision A: Tighten wording. Decision B: Add a validity follow-up to every item, and agree "existence assertion only counts when no validity evidence exists." Choose B. Rewording doesn't fix layer problems. One more thing mandatory: Rewrite — turn this incident's pattern into a new validity assertion so the checklist catches it next time.

What This Design Buys

Three Transformations

Quantified Efficiency : "Room silence" items disappear (after rewrite, every item gets an on-the-spot answer); "Untestable" separate counting auto-generates improvement to-dos.

Capability Downward Transfer : New hires can facilitate this checklist — criteria written on the table, not dependent on veteran tribal knowledge.

Pattern Upgrade : From "post-incident verbal postmortem" to "post-incident rewrite one assertion" — postmortem shifts from one-off act to checklist iteration mechanism.

Selection: How to Choose Among Four Checklist Types

Steal the default: Five Rulers Concise + Existence Baseline + Key Items Sample Validity; Reject "80-Item Mega-Table" and "Existence Only."

Scenarios Not to Copy, Four Pits

Applicable / Not Applicable

Applicable: Teams with stable delivery cadence, can reserve 30 mins before each delivery; mechanisms already exist, need to confirm they actually work; team willing to treat "Fail" as normal outcome.

Not Applicable: Scenarios shipping 10+ times a day, no fixed delivery points; starting from zero, mechanisms not even built yet (build mechanisms first, not checklist first); culture where Fail gets blamed — checklist will instantly degrade to all-pass. If Fail gets you scolded, this checklist will die. It's not a liability shield; it's a problem catcher.

Pit 1: Items Written as Adjectives

Temptation: adjectives are quick to write and look professional. Wrong antidote: moderator ticks "Yes" for everyone. Right antidote: every item must be pointable to a Yes/No.

Pit 2: Recording "Untestable" as "Pass"

Most common checklist leak. Untestable must be treated equal to Fail, and counted separately.

Pit 3: Fail Has No Consequence

Added 20 items, zero consequences. First three runs all-pass, fourth run pass rate dropped to 70% — that was the real level , previous all-greens were fake.

Pit 4: No Rewrite After Incident

No matter how detailed the postmortem, if it doesn't become a new assertion, next time it slips through just the same. A postmortem that doesn't rewrite into the checklist equals no postmortem.

Action Tiers

L1: Add/remove an assertion, reword, adjust a consequence — Team self-decides

L2: Upgrade an item from existence to validity check, change gating threshold — Platform Review

L3: Change the global rule "Untestable cannot pass" — Gate Change

Summary

Checklist value isn't in item count, but in every item being on-the-spot decidable and Fail having real consequence.

Adjectives aren't criteria — turn "good/bad" into "exists/doesn't" or "exceeds/doesn't exceed."

Three states, not two: "Untestable" equals "Fail" for gating, counted separately — it's the best to-do list.

Existence is cheap but weak; validity is expensive but real — at least give key items a validity follow-up.

Five rulers × five items = 25 items, 30 minutes, two people — beyond that, all-pass is inevitable.

One incident, one rewrite — a postmortem that doesn't feed back into the checklist is no postmortem.

The only standard for a good checklist: it has caught a real thing that would have happened.
Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

AI agentevaluationmulti-agentlong taskschecklistself-assessmentself-evolutioncontext engineeringbinary assertionsconsequence-driventhree-state verdict
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.