From Demo to Production: How Our Java Agent Harness Fixes Common Pitfalls

Java teams building agents face three major gaps when moving from a demo to production: a mismatched Java‑centric tech stack, missing enterprise‑grade governance, and runtime shortcomings such as memory drift, oversized tool results, and untraceable reasoning. Existing frameworks (Spring AI, LangChain4j, AgentScope) handle model integration but leave the agent runtime unaddressed.

1. What Java‑Side Agents Lack

Tech‑stack mismatch: Backend teams use Java, but many agent examples require Python services, forcing cross‑language RPC and duplicate observability stacks.

Enterprise‑grade governance gap: Open‑source frameworks only get the model running; tracing, configuration, rate‑limiting, audit, and gray‑release must be added by the user.

Runtime blank spot: Even with a Java solution, frameworks only provide model calls. Production‑grade concerns—short‑term memory management, handling 50 k‑character tool payloads, error replay, hot‑plug tool updates, and concurrent tool consistency—are not covered.

Getting a model to run is just an entry ticket; keeping an agent stable, controllable, and explainable in production is the real challenge.

2. Design Philosophy: How We View the Agent Runtime

Four principles guide the design of the harness:

Additive over replacement: Do not replace Spring AI; attach capabilities via Advisor/ToolCallback/Observation.

Lazy compaction (read‑write separation): Write paths are fast, millisecond‑level persistence; all compression happens only when the model reads the data.

Progressive degradation: When context pressure grows, information degrades gradually from high‑precision raw text to dense summaries instead of abrupt truncation.

Safe by default: Cleanup of error paths, concurrent consistency, safe tool decommissioning, and session resume after restart are handled automatically.

3. System Architecture: Spring‑Ai‑Trip Overview

The harness sits between Spring AI and the business agent, providing a full‑stack runtime layer.

The data flow passes through the harness for each agent call, enabling memory compression, spill handling, and observability injection.

4. Short‑Term Memory: Progressive Compression

Agents need continuous memory; without it they behave as f(current_input) → output, causing three breakages:

Conversation continuity loss (e.g., order ID forgotten after several turns).

Decision‑chain loss (intermediate conclusions disappear).

Action‑chain loss (tool results vanish, causing duplicate calls).

Memory is the core differentiator between an agent and a simple API wrapper.

Typical sliding‑window memories (Spring AI MessageWindowChatMemory, LangChain ConversationBufferWindowMemory) drop the oldest messages based solely on position, leading to goal drift when early important information is evicted.

Our approach keeps every piece of information, progressively degrading it: high‑precision raw text → high‑density summary, preserving a continuous, readable history.

We borrow three ideas from Claude Code’s context management:

Never discard information outright.

Structure summaries instead of free‑form “summarize”.

Provide explicit retrieval paths for compressed data.

4.1 Structural Defect of Common Approaches

Fixed‑size windows force a hard cut‑off; the eviction rule does not consider information value, causing critical early facts to disappear.

4.2 Three‑Layer Defense

Layer 1 – Full persistence: Every message is appended to storage without deletion, enabling replay, audit, and cross‑instance session continuation.

Layer 2 – Tool‑result micro‑compression: Large tool outputs are stored on disk; only a short preview (default first 2048 characters) plus a pointer is injected into the model context. Example: a 3000‑token tool result is reduced to a 10‑token placeholder with an evidence‑id for later retrieval.

Layer 3 – LLM‑driven structured summary: When the token budget is still exceeded, the early conversation is folded into a nine‑section structured summary, with higher‑priority sections (P0) protected from removal.

Summaries are injected as <system‑reminder> tags, preserving intent and state across many rounds.

4.3 Service‑interruption Repair

If a tool call is pending when the process restarts, a “dangling” message violates the LLM API protocol. The harness detects and removes such orphaned messages during session reload, ensuring safe continuation.

4.4 Dynamic Budgeting

Instead of a fixed percentage of the context window for history, we compute the effective window as:

effective_window = context_window - reserved_output(8000) - safety_margin(3000)
fixed_overhead = system_prompt + tool_schema + current_input
history_budget = effective_window - fixed_overhead
trigger_compression ⇔ estimated_total > effective_window × threshold(0.90)

This "subtract‑first‑then‑calculate" method prevents budget overruns caused by variable prompts, tool schemas, or user inputs.

5. Observability: Span + Event Tree

Traditional APM tracks calls and latency but cannot capture model reasoning. We model each agent execution as a tree of Span (execution intervals) and Event (model thoughts, tool inputs/outputs, errors).

Span: Nested intervals covering a full session, each round, and each tool call.

Event: Points inside a span such as Thinking (model chain of thought), tool input, tool output, and error events.

A real troubleshooting case shows a user asking why conversion dropped; the agent blamed a payment‑gateway timeout. The Span/Event replay revealed that the tool’s data only covered up to noon, while the drop happened in the afternoon, exposing a data‑coverage bug rather than a model error.

With full cognitive observability, the root cause is instantly visible: the evidence itself was incomplete.

5.2 Cognitive Observability vs. Traditional Monitoring

Our system records not only that a call happened, but also the evidence, reasoning, and conclusion that led to the answer.

5.3 Visualization Backend

A dedicated UI renders the Span+Event tree, allowing developers to see the entire reasoning chain, token consumption, and error context in one view. The final answer is just a Trace entry; the trace is the tuning entry point.

6. Spill: Large Result Protection

When a tool returns a massive payload (e.g., 50 k characters), three problems arise: token budget breach, attention dilution, and cost explosion. The harness treats such payloads like OS paging: the full result is persisted to disk, and only a preview plus a readable path is injected.

[Tool result too large, spilled to disk]
Preview (first 2048 chars): {"code":0,"data":{"list":[{"id":"...","name":"..."} ...}}
Full content saved to: spill://order‑assistant/conv‑abc/tool‑7f3a.json
Use the read‑tool with the path to fetch the needed segment.

The model decides whether to read the full content; most of the time the preview suffices.

6.1 Implementation Details

Spill threshold default 32 000 characters.

File naming follows agentName/sessionId/toolId to bind the spill file to the session lifecycle.

Concurrent spills use unique names and maintain per‑tool consistency.

AgentScope Java’s ToolResultEvictionMiddleware arrived at a similar design, confirming the approach’s validity.

7. Capability Supply: Filling Skill Gaps & MCP Hot‑Plug

Agent ability is defined by its tools. Too few tools → nothing can be done; too many tools → schema consumes massive tokens.

Skill injection: Spring AI lacks a skill layer. We provide two sources:

Built‑in classpath skills (logging, ClickHouse query, etc.).

DB‑backed dynamic skills loaded at runtime via spawn(appId, agentName), allowing new skills without code changes.

MCP hot‑plug: Tools are registered through a configuration center (e.g., QConfig). Changes take effect instantly without restart. Safety is ensured by differential refresh and reference‑counted delayed shutdown.

7.3 Built‑in Atomic Tools

File I/O, command execution, task list, HTTP calls, etc., are provided out‑of‑the‑box for reuse across business scenarios.

8. Parallel Tool Calls: Avoid Queuing

When a model decides to call several independent tools, the harness executes them on Java virtual threads, turning total latency from the sum of durations into the maximum of the parallel branches.

Result order is preserved when re‑injecting tool outputs into the prompt.

Each tool’s Span remains isolated, preventing cross‑contamination.

Spill and micro‑compression work correctly under concurrency.

9. End‑to‑End Demo: One Troubleshooting Session

User: "Help me investigate why order ORD‑2024‑0117 is stuck in 'payment pending'."

Skills and tools are loaded; the payment‑status skill is fetched and the MCP‑configured tools are hot‑plugged.

Model concurrently requests order info, payment flow, and gateway logs via virtual threads.

The gateway log returns ~40 k characters; the spill mechanism stores the full log and injects a preview with a read‑path.

The model reads the preview, decides it needs a specific segment, and fetches that part via the read‑tool.

After several rounds, progressive compression folds earlier rounds into micro‑compressed facts and a structured summary, keeping the original intent anchored.

When the final answer is produced, the entire Span+Event tree is recorded, enabling instant replay of the reasoning path.

Session termination triggers cleanup of spill files, caches, and tool connections.

The business code only needs to feed the conversation and expose tools; the harness handles memory, budgeting, anchoring, repair, observability, and cleanup.

10. Roadmap

Stronger agentic loops: plan mode, sub‑agents.

Long‑term memory across sessions.

Self‑healing and fault tolerance for dangling calls.

Prompt governance: versioned, testable, and gray‑released prompts.

Conclusion

Spring‑Ai‑Trip does not replace Spring AI; it layers on top to make a single Java agent run reliably in production. By addressing memory, spill, observability, skill injection, hot‑plug tools, and parallel execution, the harness turns demo‑only agents into production‑ready services with minimal configuration.