Integrating JEV Judgment Model into AgentScope Java for Structured Agent Decisions

Large language models enable agents to understand tasks, generate content, and call tools. As task flows grow longer, systems need concrete judgments at each execution node: which tools to expose, whether retrieved evidence supports a claim, whether an operation meets execution conditions. These judgments must feed directly into program logic to decide the next step.

JEV as a System One Judgment Model

TypeSafe's JEV (Judgment Evaluation Model) is designed for this class of needs. TypeSafe categorizes it as System One — borrowing from Kahneman's "Thinking, Fast and Slow" — emphasizing fast, focused judgments. JEV receives typed context and questions, returning options, scores, and probabilities for software branching, ranking, and routing. It decomposes a large problem into many specific judgments that code then composes.

Unlike generative models that produce text, code, or JSON, JEV is trained for typed decisions and probability calibration. Developers predefine questions and answer spaces; the model judges given material and returns program-ready results — e.g., routing a support ticket to a team, checking factual grounding of an answer, or scoring risk levels. Generative models still handle complex analysis and content generation.

Probability, Calibration, and Uncertainty

Probabilities let applications handle "uncertainty" explicitly: when evidence is insufficient, the system can retain the original path, gather more material, or request human review. TypeSafe's calibration focuses on whether a set of predicted probabilities matches observed outcome frequencies — it does not guarantee any single judgment is correct. Applications must validate thresholds and adoption strategies with their own business samples.

AgentScope Java Integration Overview

AgentScope now officially supports JEV via the agentscope-extensions-jev module. Developers can call the JEV Client and Judge components directly, or use Harness middleware for tool selection, model routing, execution guards, context governance, evidence processing, response revision, task supervision, and trace evaluation. This support clarifies the relationship between judgment and execution: the application supplies candidates, facts, and constraints; JEV returns typed results; Harness adopts results at the appropriate stage and leaves auditable execution records.

Roles: JEV, Generative Models, and Harness

Generative models : analysis, answer generation, tool call proposals.

JEV : structured judgments on specified questions.

Harness : places judgments at the right execution points, manages tools, context, budgets, and state.

They can be combined or used separately. A plain Java service doing only draft auditing can call the client or Judge directly. Deterministic rules (e.g., permission checks, refund status) remain in code; semantic judgments (e.g., whether a response omits a refund condition, whether retrieved material contains injection) go to JEV.

Structuring Requests: State and Questions

A SystemOneRequest contains shared state and ID-named questions. State holds user requests, tool results, policies, or drafts. Questions describe conditions the material must satisfy. The question ID maps to the answer in the result; one request can include multiple questions.

Example: state includes "business record: refund not executed" and "draft: refund completed"; question: "Does business_record support the refund status claimed in assistant_answer?" This yields a probability for a precise proposition. Asking "Is this answer good?" without providing policy, facts, or criteria yields non-reusable judgments.

Interpreting Probabilities, Confidence, and Uncertainty

A NoulAnswer.noul() returning 0.92 means the model judges the statement true with 92% probability — not that the operation succeeded, nor that 92% accuracy was measured on your business. Choice returns a selection and distribution from a candidate set; Score values must be interpreted against defined levels. Probabilities and confidence are decision inputs; whether they suffice depends on business thresholds and independent evaluation.

Typed results ease programmatic handling, but correct format ≠ correct judgment. Query-order and conditional-refund tools may both apply, requiring independent per-tool judgments; single-choice suits model routing. Probability thresholds must be calibrated on your business samples — do not treat example values like 0.8 as universal standards.

Above base types, JevJudge supports reusable review conditions and distinguishes PASS, FAIL, INCONCLUSIVE, and ERROR statuses, letting applications handle pass, explicit failure, insufficient evidence, and call faults separately.

End-to-End Flow: Conditional Refund Example

User request: "Check the order; if not shipped, refund and tell me the result." Judgments are placed at multiple points:

Pre-reasoning : tool selection keeps query and refund tools.

Pre-refund execution : guard checks if current evidence supports the call.

Knowledge retrieval : provides refund policy; evidence processing filters suspicious instructions while retaining useful policy that conflicts with user expectations.

Post-generation : auditor checks for missing conditions, or misstatements like "refund arrived" vs "application submitted"; revises draft if needed.

Post-invocation : trajectory evaluation checks tool usage, answer grounding, and requirement coverage.

These stages are independently enableable. Refund tool applicability ≠ conditions satisfied; one approved call ≠ business success; audit pass ≠ execution receipt from order system.

Pre-Execution: Tool Selection, Model Routing, Execution Guards

Tool Selection with Multi-Select and Bounded Context

JevToolSelectionMiddleware

evaluates each candidate tool for applicability before each reasoning round, then filters by a count limit. It can retain both query and refund tools, or explicitly withhold business tools for a greeting. Mandatory tools are kept separately and not counted toward the limit.

Default context strategy is RECENT_WINDOW: last 8 non-empty text messages, 8000-character total budget. Applications can adjust budget, send only latest user text, or use full history. Smaller windows limit judgment input but may lose early clues; evaluate against task length.

Explicitly judging all business tools inapplicable differs from judgment failure: the former keeps only mandatory tools; uncertainty, timeout, or illegal response falls back to the original tool catalog. Filtering changes what the main model sees, not tool permissions.

Phase Routing by Stage, Capability, Capacity, and Quota

Beyond per-call model selection, JevPhaseRouting supports ordered candidate pools for planning, execution, etc. The host provides model capabilities, context capacity, output reservation, and quota; unqualified candidates are excluded first, then selection occurs. Developers can explicitly start phases or let JEV judge among qualified routes, and suggest thinking levels within candidate support.

The same agent call reuses the selection; before actual dispatch, candidate conditions are re-verified; expired candidates fall back to the original model. Phases are switched explicitly by the application, not automatically by intermediate tool outputs. JevTeamPlanner can recommend authorized, available team members for a defined phase; actual dispatch remains the scheduler's responsibility.

Execution Guards Check Formed Calls

JevAutoModeMiddleware

performs pre-execution checks on specific tool calls. Example: the refund tool entered the candidate catalog, but the order status is still unknown — the actual refund call must still be checked.

In ENFORCE mode, rejected calls are not dispatched; judgment failures also reject protected calls. Business permissions, parameter validation, and idempotency remain in the original execution chain. A passing judgment only means the call may proceed into the execution pipeline — it is not an authorization or proof of business success.

During Execution: Context Governance, Memory, and Retrieval Evidence

History Compaction with Constraints and Original Material Recovery

Long tasks accumulate file reads, search results, and tool history. JevContextCompactor plugs into Harness's existing compression entry. Within the host-specified tool scope, it judges whether historical calls and results should be kept, truncated, or removed pairwise from the current context.

Recent messages, system messages, fixed constraints, and incomplete tool results have explicit protection rules; uncertain material is retained. In ENFORCE mode, compaction is adopted only after successful archiving; archive failure or insufficient reduction retains current history. Subsequent recovery reads saved messages — it does not re-execute historical tools.

Truncation keeps result prefixes, not generated summaries. The final request still passes Harness's budget check. Applications can choose file or BaseStore archiving, isolated by user, session, and agent.

Long-term memory has a separate entry: JevMemoryGate simultaneously reviews candidate facts for persistence value and conflict with existing facts. It provides admission advice; actual writes are the application's responsibility — context compaction does not automatically promote material to long-term memory.

Retrieval Processing Keeps Evidence That Corrects Premises

RAG needs more than relevance ranking. If a user asks "Will the refund definitely succeed?", a policy statement "Refund requires conditions and review" negates the premise but is critical evidence. JevEvidenceProcessor supports classification, filtering, and re-ranking of authorized material, distinguishing prompt injection, answer evidence, and premise-conflicting evidence. The host runs ACL first; JEV then processes visible material. Developers can process passages directly, expose them via the search_evidence tool to the agent, or use the Knowledge adapter to plug into the retrieval pipeline.

Retaining evidence is step one; step two checks whether the generated answer is actually grounded. These address "what material to give the model" and "whether the model faithfully used the material" respectively.

Output Time: Auditing Drafts and Revising Only What's Needed

When policy states "unused and within seven days refundable" but the agent writes "unconditional refund", the answer needs correction. Re-running the entire agent might repeat completed business operations. JevResponseMiddleware audits final text with no tool calls. In ENFORCE mode, an explicitly failed draft can enter a limited number of pure-text revisions; revision calls carry no tools and do not re-run the agent tool loop. The revised draft must pass audit to be published; insufficient evidence, audit errors, or exhausted revisions block publication of the unapproved original.

This affects streaming: revisions are not streamed token-by-token; text is emitted only after audit passes. Applications must set audit and revision budgets and handle rejection or human-escalation outcomes. SHADOW only audits the original, does not revise, and does not block release.

Content guardrails can combine with answer-quality audits; applications with existing drafts can plug into the standalone draft pipeline.

Long Tasks and Delivery: Supervision and Trajectory Evaluation

Continuous Supervision Monitors Progress and Verification Evidence

JevSupervisionMiddleware

observes during task execution whether there is substantive progress, stalling, or drift, and whether additional verification is needed. It advises continue, re-check direction, request verification, or complete verification.

The supervisor only supports OFF / SHADOW — it never stops tasks or marks them complete. Completion verification requires the host to provide verification evidence matching the current version and scope; an agent merely claiming "done" cannot form a trusted delivery conclusion. Applications can also trigger one-off task reviews on their own schedule.

Trajectory Evaluation Separates Tool Behavior from Answer Quality

JevTraceEvaluator

can evaluate historical trajectories or collect real calls via middleware. Built-in metrics cover tool selection, tool result utilization, answer grounding, scope adherence, relevance, completeness, and indirect prompt injection, plus optional deterministic tool-trajectory baselines that don't call the model.

The same definitions serve online observation and offline batch evaluation; custom metrics support business-specific standards. Results retain category, score, pass/fail, and status — avoiding conflation of skipped, abstained, and error into a single low score.

Post-hoc evaluation does not undo tool executions, nor re-run business flows to improve scores; online middleware only supports OFF / SHADOW. Its value is surfacing issues, comparing versions, and building regression samples.

Further Extensions: Code Review and Read-Only Browser Tasks

Structured judgment also fits specialized tasks with clear candidates and evidence boundaries. JevCodeReviewer: phases risk screening on diffs or source snapshots, then locates evidence from parsed real regions, outputting mechanism, severity, and items for re-check. Only OFF / SHADOW; does not auto-publish reviews or approve merges; absence of findings ≠ proof of code safety. Used to organize review leads; complex causal analysis and test verification still need follow-up. JevBrowserNavigator: selects read-only navigation, scroll, or wait from authorized page candidates, feeding the read_browser tool into the agent. The host provides an exclusive browser session and an independent completion verifier; page changes require re-observation. Only independent verification yields VERIFIED; model choosing DONE alone does not constitute completion. Suited for controlled tasks like help-center and policy queries.

For business-entry multi-label routing, conditional demands, and customer-service reply commitment checks, candidate classification and service review can be composed directly without building a full agent first.

Practical Integration: Start with One Use Case

1. Add Dependency

Use Java 17+; replace ${agentscope.version} with a version containing these APIs, keeping all modules consistent. For direct client, Judge, or evaluator use, add the extension module:

<dependency>
  <groupId>io.agentscope</groupId>
  <artifactId>agentscope-extensions-jev</artifactId>
  <version>${agentscope.version}</version>
</dependency>

After obtaining API credentials from TypeSafe, set TYPESAFE_API_KEY in the Java process environment. This credential is used by the JEV judgment interface; the agent's original generative model keeps its own config. The client reads TYPESAFE_API_KEY then JEV_API_KEY if not explicitly set; neither valid → client cannot be built.

2. First Real Judgment

Standalone Java entry point (runnable from IDE after adding the dependency). It calls the real API, requires credentials, and incurs usage.

import io.agentscope.extensions.judge.jev.JevClient;
import io.agentscope.extensions.judge.jev.JevRetryPolicy;
import io.agentscope.extensions.judge.jev.NoulAnswer;
import io.agentscope.extensions.judge.jev.NoulQuestion;
import io.agentscope.extensions.judge.jev.SystemOneRequest;
import java.time.Duration;
import java.util.Map;

public class JevQuickStart {
  public static void main(String[] args) {
    var client = JevClient.builder()
      .baseUrl("https://api.typesafe.ai")
      .model("jev-latest")
      .timeout(Duration.ofSeconds(5))
      .retryPolicy(new JevRetryPolicy(2, Duration.ofMillis(500)))
      .build();
    var request = SystemOneRequest.builder()
      .state(Map.of(
        "business_record", "尚未执行退款",
        "assistant_answer", "退款已经完成"))
      .question("supported", new NoulQuestion(
        "Does business_record support the refund status claimed in assistant_answer?",
        null))
      .build();
    var result = client.systemOneBlocking(request);
    var answer = (NoulAnswer) result.answers().get("supported");
    System.out.println("记录支持草稿的概率：" + answer.noul());
  }
}

Successful call prints the probability for supported. The input deliberately contains a factual conflict, expecting a low support probability, but the exact value comes from the live model — do not hardcode it as an acceptance criterion. Change the draft to "refund not yet executed", compare the two outputs on the same question, then add real business samples for evaluation. systemOneBlocking suits this standalone entry; reactive services should compose client.systemOne(request) returning a Mono to avoid blocking the event loop. The client validates answer types, question sets, and probability fields; request or response failures throw JevException — do not interpret interface errors as zero probability.

The 5-second timeout is per request; retries add to total latency. Higher-level JevExecution can budget the entire judgment logic including retries and backoff.

3. Turn Probabilities into Reusable Review Results

If the application needs to decide whether a draft proceeds, reuse the client and define a JevJudge condition:

import io.agentscope.extensions.judge.jev.JevJudge;
import io.agentscope.extensions.judge.jev.NoulQuestion;
import java.time.Duration;
import java.util.List;
import java.util.Map;

var definition = new JevJudge.Definition("refund-answer-v1", List.of(
  new JevJudge.Criterion(
    "supported",
    new NoulQuestion(
      "Does business_record support the refund status claimed in assistant_answer?",
      null),
    true, 0.2, 0.8)));
var judge = new JevJudge(client, Duration.ofSeconds(2));
var review = judge.judge(
  Map.of("business_record", "尚未执行退款", "assistant_answer", "退款已经完成"),
  definition);
var observed = review.doOnNext(result -> {
  System.out.println("评审状态：" + result.status());
  System.out.println("条件明细：" + result.findings());
});
// Reactive services return or compose observed; standalone CLI can call observed.block().

true

means the condition is expected to hold; in the example, ≤ 0.2 → FAIL, ≥ 0.8 → PASS, middle → INCONCLUSIVE; request faults → ERROR. These thresholds merely illustrate status branches — calibrate with business samples.

The Judge returns a result; it does not auto-publish or revise. The application should: on PASS continue post-publish checks; on FAIL revise or re-check; on INCONCLUSIVE gather more evidence; on ERROR apply failure handling. When changing criteria or thresholds, bump the definition version for offline regression comparison. A Mono must not be subscribed multiple times for logging, or it will re-evaluate.

4. Attach Judgments to Existing Harness

When judgments must run automatically before each reasoning round, use middleware. Harness integration requires an additional same-version dependency:

<dependency>
  <groupId>io.agentscope</groupId>
  <artifactId>agentscope-harness</artifactId>
  <version>${agentscope.version}</version>
</dependency>

Using the previously created client, model, and toolkit (your existing generative model and tool registry — tool names and schemas must be registered; middleware filters, does not create tools):

import io.agentscope.extensions.judge.jev.JevExecution;
import io.agentscope.extensions.judge.jev.example.JevToolSelectionMiddleware;
import io.agentscope.harness.agent.HarnessAgent;
import java.time.Duration;

var options = new JevExecution.Options(
  JevExecution.Mode.SHADOW,
  Duration.ofSeconds(2),
  "order-tools-v1",
  (ctx, record) -> System.out.println(record));
var selector = JevToolSelectionMiddleware.builder(client)
  .execution(options)
  .maxTools(2)
  .confidenceThreshold(0.8)
  .rejectionThreshold(0.2)
  .build();
var agent = HarnessAgent.builder()
  .name("order-support")
  .model(model)
  .toolkit(toolkit)
  .middleware(selector)
  .build();

Thresholds here are for demonstration; in SHADOW the main model still receives the original tool catalog. Production observers should return quickly, associating judgment version, latency, and advice with the current run. Default middleware mode is OFF; SHADOW logs advice; ENFORCE adopts advice per use case. Direct Judge/API calls trigger reviews explicitly, not implicitly controlled by a global mode.

To verify wiring, run the existing agent with "check order", "refund if not shipped", and a plain greeting; observe whether records show corresponding tool suggestions. In SHADOW the actual tool catalog should stay unchanged; after calibration, switch to ENFORCE, rebuild middleware and agent, and confirm the tools actually passed to the model. The original agent invocation remains unchanged; registering middleware does not itself execute a task.

5. Spring Boot Auto-Configured Client

Spring Boot projects can use the Starter for client integration:

<dependency>
  <groupId>io.agentscope</groupId>
  <artifactId>agentscope-jev-spring-boot-starter</artifactId>
  <version>${agentscope.version}</version>
</dependency>

In application.yml:

agentscope:
  jev:
    api-key: ${TYPESAFE_API_KEY:}
    base-url: https://api.typesafe.ai
    model: jev-latest
    timeout: 5s
    retry:
      max-retries: 2
      initial-backoff: 500ms

The Starter creates a JevClient bean when no custom one exists. Inject via constructor to reuse the earlier request and Judge definitions:

import io.agentscope.extensions.judge.jev.JevClient;
import io.agentscope.extensions.judge.jev.SystemOneRequest;
import io.agentscope.extensions.judge.jev.SystemOneResult;
import org.springframework.stereotype.Service;
import reactor.core.publisher.Mono;

@Service
public class DecisionService {
  private final JevClient client;
  public DecisionService(JevClient client) {
    this.client = client;
  }
  public Mono<SystemOneResult> decide(SystemOneRequest request) {
    return client.systemOne(request);
  }
}

The Starter configures the client only — it does not auto-register tool selection or enable ENFORCE for all agents. Middleware must still be explicitly assembled; to customize the client, provide a JevClientBuilderCustomizer bean. agentscope.jev.enabled=false disables the Starter's auto-creation; dependent business components must adjust accordingly.

6. Service Config and Keyless Trial Entry

AgentScope Service users can configure tool selection, guards, routing, content/quality audits, evaluation, compaction, supervision, code review, retrieval, and browser via Session's agentOverrides.jev. Keys are supplied by the deployment environment; retrieval sources, model catalog, browser sessions, and verification evidence still require host integration.

To see the flow run without keys, agentscope-examples/jev provides offline examples covering retrieval with answer revision, compaction with archiving, phase routing, supervision, and trajectory evaluation. Build and run per the example guide; scripted models and synthetic judgments verify the pipeline, not real model performance.

Evaluate with Real Tasks Before Committing

Adding judgments adds requests and latency. Fewer tool descriptions don't guarantee faster end-to-end; more audits don't guarantee higher task success. Compare the original flow against the enhanced flow on identical inputs, measuring judgment quality, actual dispatch behavior, and final business outcomes.

For refund guards, track both erroneous passes and legitimate refunds blocked; for answer revision, observe final correctness and added latency; for compaction, check key fact retention and downstream answer quality. Preserve errors, abstentions, and skips; separate judgment latency from overall latency; don't record missing usage as zero cost.

Start with SHADOW to accumulate samples, then decide per use case whether to ENFORCE, and verify fallbacks meet business requirements: tool selection and routing failures keep original flow; protected tool check failures reject dispatch; pre-publish audit failures block text release.

Official docs provide full evaluation guides, budgeting, and observability. Share your use cases, missing capabilities, and real evaluation results in Issue #3251 — whether custom middleware, complete examples, or attempts that yielded no benefit, all help improve AgentScope Java's judgment pipeline.