openJiuwen and Ascend Enable Agent “Compute‑Affinity”: Halve First‑Token Latency, Cut Inference Storage by 25%

When an Agent receives a task, it may spend minutes or hours planning, invoking tools, reading files, and iterating; multiple Agents may collaborate, causing the KV Cache generated during inference to grow dramatically and increase latency.

Traditional inference engines only see the request and the cache, lacking visibility into the Agent’s task state—whether a sub‑task has finished, a session is paused, or a session is about to resume—leading to inefficient cache management.

The core gap is the missing semantic channel between task state (Agent) and compute resources (engine). openJiuwen, a collaborative effort by Huawei’s 2012 Lab, Cloud, Compute, and Terminal teams, builds this channel through a full‑link coordination mechanism that translates Agent task states into executable scheduling signals for the engine.

Agent Hint: Semantic Contract Between Agent and Engine

openJiuwen’s solution is simple: when an Agent’s state changes, it immediately sends a Hint to the inference engine. The Hint is not an extra API call but a semantic payload indicating the session ID, parent session ID, and the intended cache actions (evict, offload, prefetch). The engine then performs the corresponding cache operation, forming a closed‑loop lifecycle for KV Cache.

{
  "agent_hint": {
    "session_id": "sub-1", // which session
    "parent_session_id": "main-0", // parent task
    "context_management": {
      "edits": [{"type": "offload", "target": "tools"}]
    }
  }
}

Swarm Task Walk‑through

Task planning : The Leader decomposes the task; shared prompts and tool definitions are cached for reuse by all sub‑Agents.

Sub‑Agent calls : Before each sub‑Agent activation, JiuwenSwarm issues a prefetch signal to bring the previous round’s cache back into HBM.

Tool execution : When a sub‑Agent waits for an external tool, its KV Cache is offloaded from HBM, freeing fast memory; once the tool returns, a prefetch restores the cache before the next inference step.

Context compression : Long‑running tasks may compress or trim their context; removed parts trigger an unload signal, moving the cache to lower‑cost storage.

Session end : Completed sessions have reusable cache moved to low‑cost storage, while unnecessary cache is evicted; all resources are reclaimed at the swarm’s final boundary.

Session resume : Even finished sessions can be re‑activated; a prefetch brings stored cache back to HBM before continuation.

Three‑Layer Coordination

SAM (Session‑Aware Manager) upgrades the engine’s prefix cache to be session‑aware, preserving active session prefixes and quickly reclaiming finished sessions.

SPM (Session‑Aware Pooling Manager) extends session semantics to the distributed cache pool (e.g., Mooncake), keeping active sessions alive and prefetching caches for resumed sessions.

Ascend interconnect leverages the high‑speed Lingqu bus to move cache across NPU HBM, Kunpeng CPU DDR, SSD, and remote pools, ensuring that hints trigger precise placement while the bus provides rapid data flow.

Performance Gains

Using the SWE‑bench Verified dataset, openJiuwen compared runs with and without compute‑affinity across bug‑fix, feature‑development, and code‑refactor scenarios with ten concurrent users. Results show:

First‑token latency (TTFT) reduced by 57.4%.

End‑to‑end latency reduced by 27.61%.

Prefix‑cache hit rate increased by 33%.

Peak pooled‑cache usage decreased by 25.24%.

Conclusion

By making the engine understand Agent task states, openJiuwen’s compute‑affinity transforms cache management from a generic LRU approach to a semantic, session‑aware scheduler. This yields higher cache utilization, more stable multi‑turn inference, faster session recovery, stronger multi‑Agent capacity, and lower overall cost.

With the same hardware and tasks, first‑token latency drops over 57% and inference storage peaks shrink by about 25%.