Goal Engineering: Define Agent Goals by Starting with Acceptance Criteria
The article introduces Goal Engineering — designing AI agent goals by first determining what evidence human reviewers need to accept results — illustrated through a urllib3 2.x upgrade case study across three services showing how to structure conclusions with evidence, next steps, review boundaries, and task state management for waiting and deferred work.
Start with Acceptance, Then Define Goals
The author illustrates Goal Engineering using a concrete scenario: upgrading three Python services from urllib3 1.x to 2.4.0. An agent returns a report stating "Order service: old parameter not found, recommend direct upgrade." But the maintainer must first know how deep the agent looked. If the order service uses a shared client, retry parameters may live inside that client — invisible in the business repo. The maintainer would then have to manually check versions and source code, pulling work back from the agent.
LangChain's Harrison Chase and Actively AI's Mihir Garimella and Anshul Gupta co-authored Goal Engineering , which focuses on scoping an agent's work so that knowledgeable people can inspect, correct, and reuse those corrections. Applied to the upgrade assistant, the author suggests working backwards from acceptance: the maintainer must be able to judge what needs changing, what's already adapted, and what lacks evidence. What evidence must be in front of them to decide the next step? Clarify that first, then assign the agent's division of labor, materials, and follow-up.
urllib3 Change Verification
urllib3 1.26.0 deprecated method_whitelist; 2.0.0 removed it. A service might have: Retry(method_whitelist={"GET"}) The author verified both versions in the same Python process. With pip-bundled 1.26.20, the line creates a policy and emits a deprecation warning. With separately installed 2.4.0, it raises TypeError. Changing to Retry(allowed_methods={"GET"}) works on both versions.
Thus a report saying "local run passes" is insufficient — the maintainer must know which version ran. The author also checked the retry policy's decision function: with status 503 and a Retry-After header, the new syntax allows retrying GET and excludes POST on both versions. This validates policy construction and that input set, but not full network requests. Full upgrade viability still requires checking other compatibility changes and running regression tests.
Three Services, Three Conclusions
Payment service needs change. Directly uses old parameter to create Retry; report includes call site, target version error, and post-replacement verification. Maintainer can review the diff directly.
Reconciliation service already adapted. Currently uses allowed_methods; report cites source location for the relevant version. Maintainer confirms no change needed for this parameter migration.
Order service missing version info. Business code calls shared client; deployed client version unknown. Report lists discovered entry points, asks owner for actual deployed version, then continues verification against that implementation.
Each conclusion carries evidence and a next step, so the maintainer knows whether to review a diff, confirm pass, or gather more info. When the order service lacks a version number, the report returns a concrete blocker while the upgrade decision stays in the backlog. If the version is already recorded where the agent can read it, the agent fetches it itself.
Rewritten Goal
The vague "check what needs changing" becomes:
Track impact assessment for upgrading three services from current version to urllib3 2.4.0. This round handles retry parameter migration. For each service, locate the actual retry implementation , recording relevant dependency versions, call sites, and verification results. Changes needed: submit patches for owner review. Already adapted: provide evidence. When information is missing, record what's missing and who to confirm with. Resume investigation after receiving info. When owner defers, remember reason and resume condition. After all three services produce evidence-based recommendations confirmed by owners, conclude this investigation round. Items waiting for info, review, or deferral remain as todos. Merge and release timing decided by owners.
This defines outcomes and boundaries; specific file search order and check sequence can be left to the agent. When a shared client is found, the agent can append investigation tasks. When order service lacks info, the agent can process payment service first. Actions are judged useful only if they give an upgrade conclusion evidence or make the next step explicit.
Additional considerations may emerge only when the maintainer sees results. Example: since the new parameter works on both versions, could we rename the parameter first, release, then upgrade major version? That isolates blame if issues arise. Such considerations can't all be spelled out upfront.
Organize by Review Type (Motions)
After goals are clear, decide work allocation. Three repos → three agents is intuitive. But if all three do the same parameter migration, the maintainer may prefer reviewing that class of changes in one place and stating common requirements once.
Actively AI faced a similar issue. They initially assigned one sales agent per prospect. Prospect data was centralized, but when sales reviewed results, they had to switch between outreach emails, prospect replies, and deal strategies. Feedback on one email often applied to similar emails for other prospects, requiring manual propagation.
They reorganized agents by work type, called motions — e.g., proactive outreach, early-stage deal advancement — which can span prospects. Context-gathering agents still organize by prospect. Sales can review one work type at a time, and feedback reuses more easily across that work type.
The transferable insight: separate two relationships. Prospect data organizes around prospects; review experience accumulates around work types. Storage boundaries and human review boundaries can differ. Exact agent count and process layout are separate decisions.
Applied to the upgrade assistant: each repo keeps its dependency manifests and deployment records; same-type migration reports go to the person familiar with the shared client for centralized review. When they note "should verify actually installed version" on payment service, that requirement applies to subsequent repos. Each repo's release schedule stays with its owner.
Whether to centralize depends on whether a given comment typically applies across multiple tasks. If multiple repos repeatedly receive the same feedback, centralized review is worth trying. If business rules differ greatly, keep per-repo review requirements; don't propagate one repo's decision blindly.
The original paper also stresses that some judgments live only in experts' heads until they see concrete results. The "rename parameter first, then major upgrade" consideration is one example. Review interfaces must make it easy to capture such feedback.
Example: clicking "needs change" shows corresponding version, source, and verification result. When maintainer selects "recommend direct upgrade" and leaves a comment, the system saves that comment together with the evidence it addresses. Later, anyone can see why the conclusion changed and judge where else the comment applies.
These designs affect whether people bother leaving comments. If every time they must dig through chat, check deployments, find source — they'll just want to finish the current change.
Clarify Review Comment Scope
Assume order service owner fills in version info and leaves a comment:
Production uses old shared client. This repo freezes Friday, resume Monday. For future shared client migrations, please attach deployed version and corresponding implementation.
This comment contains repo fact, current decision, and general requirement. Before saving, split each sentence by applicability and validity period so later tasks know when to read it.
Production uses old shared client → Order service deployment record, with specific version, environment, confirmation time; used for this investigation
Freeze Friday, resume Monday → Order service upgrade progress log; used when scheduling resume task
Attach client version and implementation in future → After team confirmation, write into shared client migration review requirements; used for subsequent same-type tasks
Deployment version changes; resume task must confirm updates. Friday freeze affects only this order service arrangement; other repos proceed. The "attach actual version and implementation" requirement, once team-confirmed, applies to other repos' same-type tasks. If the whole comment stays only in order service chat, next shared client review the maintainer may repeat themselves.
The general requirement can be made more complete, preserving its applicability conditions and original rationale:
For shared client dependency migrations, reference the actually deployed version's implementation. When business repo search finds no old parameter, continue verifying client internals; retry config may be there.
These can live in existing deployment records, upgrade todos, and review docs. Next repo, the context-preparation program fetches that repo's dependency materials plus applicable review requirements. Reuse depends on what the task actually reads at start. Saving chat alone doesn't complete this step.
If the next report has the same issue, trace through task records. Sometimes requirements are stored but not read at task start. Sometimes read but agent checks client latest branch instead of deployed version. First needs material reading fixed; second needs checking how the agent understood and executed the requirement.
Put Waiting into Task State
One piece remains: owner says resume Monday — who continues? Harrison Chase's From tasks to goals also addresses this. Delegating a responsibility long-term to an agent requires remembering prior decisions, scheduling time- or event-based triggers, and handling pause and stop.
Implementation: keep one record per upgrade, storing current state, existing evidence, missing info, owner, and resume conditions. On Friday deferral, update record. On Monday trigger, read record, confirm still needed, then continue agent investigation.
Distinguish two wait reasons. Just waiting for version number — resume on receipt. Owner explicitly requires Monday — save new materials first, resume at agreed time. If owner completes upgrade and closes record before Monday, Monday's task sees it ended and skips.
Both scheduled triggers and human replies point to the same upgrade record. Only one task may claim a given todo at a time ; claiming atomically sets state to "running" so two tasks can't both execute. Another trigger seeing it running skips. Otherwise version info just arrived and scheduled check also fires, possibly launching duplicate investigations and sending two reports.
"Waiting for info" must attach to specific questions. Order service missing version can pause that item while continuing payment service patch review. When version unchanged and owner unresponsive, next scheduled trigger keeps waiting state. Otherwise "continuous follow-up" easily becomes asking every half hour.
Thus a trigger is merely an opportunity to check. Whether to proceed is decided by current state and resume conditions together. Agent can re-plan investigation steps; owner's prior deferral or closure decisions are preserved in the task record.
Validate Improvements with New Tasks
Finally, verify the arrangements help. The original order service report was already flagged by maintainer; re-running only proves this fix works. To test whether the review comment transfers to new tasks, pick several repos not involved in prior fixes, deliberately preserving three conditions:
Deployed shared client already uses new parameter → Cite deployed version and implementation; confirm this item needs no change
Client latest branch adapted, deployed old version still uses old parameter → Locate issue by deployed version; explain optional handling: upgrade client or patch old version
Only business repo visible, cannot confirm deployed version → Leave discovered entry points and specific problem; wait for info
In the second case, if agent grabs latest source it may wrongly conclude "already compatible." In the first, deployed version already adapted; blindly copying payment service's fix would create unnecessary changes. Both cases together reveal whether the agent judges based on the current service's actual situation.
Recently researching Codex modifying PyPA build, the author encountered a verification scope problem. Two patches passed the same ten tests; adding a same-process thread probe revealed one patch redirected unrelated thread output. Verification example choice directly shapes result judgment. The deliberate "latest fixed, deployed not" repo tests whether the review comment survives a fresh check.
For comparison, run original task spec and enhanced goal+review requirements on these new tasks, keeping input materials, tool permissions, and model settings consistent. Check not only correctness but also: how much extra material maintainer had to fetch, whether affected calls were missed, how many changes were actually unnecessary. Repeat when results fluctuate; don't judge mastery from a single pass.
A misleading metric: goal is making it easier for owners to decide upgrades, so PR count alone cannot represent progress. Already-adapted services submitting fewer unnecessary diffs is doing it right. Services still missing version info that clearly state the problem and wait for materials are more useful than emitting an unsupported "can upgrade."
The original paper predicts prompt engineering and context engineering will disappear; goal engineering will remain. That's the authors' future view. In today's upgrade assistant, even with goals clarified, how materials are supplied and how opinions enter the next task still need design. From this example, goal design can work backwards from human judgment: first see what results people need, then arrange the work that supports those results.
If you have work repeatedly sent back to agents, start from the latest comment. Identify the material you fetched and the judgment you made, write it as a clearly scoped requirement, then test on a task with different conditions. If the same extra fetch is needed next time, have the agent compare execution logs and context to diagnose whether the requirement wasn't read or was read but not executed properly. Adjust goal, material reading, or execution based on findings.
References
Harrison Chase, Mihir Garimella, Anshul Gupta, Goal Engineering: How to Design Goals (https://www.langchain.com/blog/goal-engineering)
Harrison Chase, From tasks to goals (https://www.langchain.com/blog/from-tasks-to-goals)
urllib3, 2.0.0 and prior changelog (https://github.com/urllib3/urllib3/blob/2.0.0/CHANGES.rst)
Xiao G, Prompt and Context Engineering Dead, Goal Engineering Arrives! (https://mp.weixin.qq.com/s/c5fHPNQbGiUCscpTTb865g)
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
