JEMM-27B: Open-Source Multimodal Agent Decision Model Hits 93.3% Tool Accuracy

Overview

JEMM-27B is a newly open-sourced multimodal decision model built on the Qwen base with 27 billion parameters. Released under the Apache-2.0 license, it is positioned as "Like Jev, but multimodal and open-weight." The model serves as a high-speed decision component inside agent systems, handling frequent, low-level choices such as tool selection, action ranking, and conditional judgments.

The Jev Paradigm: Separating Planning from Fast Decisions

Modern agents often rely on large language models for both complex planning and repetitive step-by-step decisions. The article explains that invoking a full LLM for every minor choice — click a button, open a link, type text, go back — accumulates latency and cost. Jev introduced a different architecture: a lightweight "quick decider" that only answers "Which action is more suitable?" given a set of pre-generated candidates. The large model handles planning; the decision model handles high-frequency micro-judgments.

JEMM Extends Jev with Visual Input

While Jev processes only text, JEMM adds image understanding. For browser agents, the current page screenshot is part of the task state. JEMM can accept up to four screenshots alongside the task instruction and candidate actions, allowing it to see button positions, layout, pop-ups, and icon meanings that text or DOM alone cannot convey. This visual capability directly benefits GUI agents and web automation.

Benchmark results provided by the project show that on screenshot-based web tasks, accuracy rises from 36.0% to 54.6% , an improvement of 18.6 percentage points .

Tool-Calling Accuracy and the Referee Role

JEMM also receives specialized training for tool selection — a frequent bottleneck in agent execution. The model treats tool choice as a decision problem: the agent proposes candidate tools (search, browser, code execution, file handling, database query, etc.), and JEMM picks the most appropriate one. Reported tool-calling accuracy improves from 88.7% to 93.3% .

The article emphasizes that JEMM acts as a "referee": candidates are generated by the agent, JEMM only selects, judges, or scores them . It does not replace the general-purpose LLM.

Latency Gains

Speed is critical because a single task may invoke the decision model dozens or hundreds of times. The project tested five text-request configurations and found JEMM's latency lower than Jev in all cases. The median per-question latency is approximately 0.3 seconds , and aggregated latency drops by 31.2% .

Integration and API Usage

JEMM is distributed via GitHub and can be installed and served with the following commands:

pip install git+https://github.com/ypcypc/JEMM
python -m jemm.serve --adapter MaestroYan/JEMM --port 8790

After starting, agents call the /v1/systemone endpoint. The request payload contains the current state and a questions object that defines the decision type ( choice, noul for judgment, score for scoring) and the criteria for each candidate. An example for a weather query:

{
  "state": "User: Will it rain in Paris tomorrow?",
  "questions": {
    "tool": {
      "type": "choice",
      "criteria": {
        "get_weather": "Weather forecast",
        "search_flights": "Flight search",
        "none": "No tool applies"
      }
    }
  }
}

For web agents, up to four screenshots can be included in the request. The response returns the selected action, judgment, or scores, which the agent then executes.

Place in the Agent Stack

The article summarizes the workflow: Task → Candidate Actions → JEMM Decision → Tool Execution → Result . By offloading high-frequency selection, judgment, and scoring to a specialized 27B model, existing agent frameworks can reduce latency and improve reliability without replacing their core planning LLM.