TLS AgentLoop: Making Multimodal Agent Calls Transparent and Traceable

Why Execution Records Alone Are Insufficient

Traditional call chains answer "how a task executed": which steps triggered, latency per stage, and error locations. For pure text tasks this often suffices. Multimodal tasks add a second question: "what content was the execution based on?"

Example: A developer asks an agent to fix a page layout from a screenshot. The model call and code edit both show success, yet the page still looks wrong. The developer must verify: was the uploaded screenshot correct? Are key regions clear? Does the prompt accurately describe the expected result? Do the model output and tool edits correspond to the screenshot's issues?

If logs only contain attachment IDs, the original file must be fetched elsewhere. If images are stored separately, linking them to a specific session and turn is hard. Embedding large Base64 blobs in logs creates three problems:

Data integrity : Base64 mixed into logs or trace spans can cause truncation, making logs/traces incomplete — especially for video and audio.

High cost : Indexing and storing Base64 as generic trace/log fields is expensive.

Low usability : Reading, managing, and correlating multimodal files with traces/logs is inefficient.

Bringing Media Back into Sessions and Call Chains

TLS AgentLoop reuses the existing Session–Trace–Span hierarchy: Session links multi-turn conversations, Trace represents a single request or task, Span records model calls, tool executions, and other stages. Multimodal content appears in the corresponding message and call nodes, not in an isolated attachment list.

List Discovery: Find Tasks Containing Media

In Trace and Session lists, records with multimodal input display a media indicator. Developers can combine this with status, latency, and token data to quickly filter records needing deeper inspection.

Drilling into details, the left call tree continues to flag spans or traces that contain media, preserving the existing navigation path even across multi-turn, multi-step tasks.

In-Place Preview: Put Images Back into Message Context

In the formatted input/output view, images preview directly and can be enlarged on demand. For supported media types, the frontend also plays audio and video. Text and media stay in the same message context, so developers can compare prompt, model answer, and execution info without stitching together screenshots, logs, and call chains.

Session Replay: See How the Problem Evolved

In Session details, multi-turn tasks show each turn's input and output continuously, with the ability to jump into the associated Trace for execution details. Media stays attached to the turn and text it belongs to. For team collaboration, this reduces the overhead of forwarding screenshots, pasting logs, and adding context — everyone discusses around the same session and call chain.

Troubleshooting Path for a Single Image-Text Request

When an agent task with an image yields an unexpected result, follow three steps:

1. Verify Input

Expand the relevant model call and inspect the captured screenshot and text requirements in the input area. Check three things: is the screenshot the expected image? Are key regions clear? Does the prompt explicitly state the desired outcome? If the input itself is missing or ambiguous, fix the input composition and prompt first rather than blaming model capability.

2. Cross-Check Execution

If input matches expectations, examine model output, tool parameters, and tool results. Does the model's suggested fix address the screenshot's issue? Did the tool edit the correct file? Did any exception occur during execution? Did subsequent calls continue processing based on the tool result? The image provides task background; the trace provides execution steps; together they prevent stopping at "result is wrong."

3. Track Feedback

For tasks requiring multiple follow-ups, return to the Session view to see whether user feedback appears in the corresponding turn and whether the model adjusted its answer accordingly. Important boundary: a record not showing an image does not prove the model never received it. Media visibility depends on collection switches, plugin coverage, upload success, access permissions, and network conditions.

Implementation Approaches

Different teams source multimodal data differently:

From agent runtime attachments.

From customer-owned TOS buckets.

From publicly accessible image, audio, or video URLs.

TLS AgentLoop provides ingestion paths for each scenario so media becomes viewable, correlatable call context.

TLS Plugin Internal Multimodal Capability

For local attachments produced or received during agent runs, the TLS plugin/SDK identifies multimodal content at collection time, handling media reading, upload authorization, object storage, and reference reporting.

Using DSH image collection as an example: the plugin detects image attachments in model input, reads file content, and extracts MIME type, file size, and content hash. It then requests an upload address via TLS attachment capability, obtains a TOS object identifier and temporary upload URL, and uploads the image to TOS. At trace report time, the plugin writes a media reference (object location, media type, size, hash) in the original message position instead of embedding Base64. TLS records the trace, message structure, and media reference; TOS stores the binary content. This suits customers who want the collection plugin/SDK to manage the media pipeline.

After integration, developers can view images directly in AgentLoop sessions and call chains and troubleshoot against prompts, model outputs, and tool results.

Associating Existing TOS Private Multimodal Resources

If customers already store images, audio, or video in their own TOS, TLS AgentLoop can link those resources to the call process without re-uploading. The business system or collection SDK includes media object references (TOS object location, media type, file size, content hash) when reporting traces. TLS saves "this message references which media object," not the media file itself.

When developers view session or trace details, the frontend requests a temporary access URL from the media reference, reads the content from TOS, and renders the preview. This preserves the customer's storage and permission management while placing media in the call context.

This fits customers with existing multimodal asset management — e.g., images, recordings, and video clips unified in a private TOS bucket. They only need to associate those resources with agent call records for debugging, case archiving, and later evaluation.

Pre-integration checks: resource access permissions, temporary URL generation method, retention period, and cross-system permission boundaries. Traces store traceable media references, not long-lived signed URLs.

Compatible with GenAI Standard Multimedia Messages

Beyond TLS custom references, the AgentLoop frontend recognizes GenAI-spec multimodal messages. For data carrying publicly accessible media URLs, the frontend uses the media part's type, modality, and mime_type fields for direct rendering. Audio, video, PDF, or multi-file scenarios can follow the OpenTelemetry GenAI multimodal message spec and examples (https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/non-normative/examples-llm-calls.md#multimodal-inputs-example): each file maps to a parts element with its transport mode, media type, and file URL. TLS frontend provides image preview, audio/video playback, or file open entry per supported range.

Example Attributes.gen_ai.input.messages structure:

[
  {
    "role": "user",
    "parts": [
      {
        "type": "text",
        "content": "请结合这段录音、视频和 PDF，总结主要内容。"
      },
      {
        "type": "uri",
        "modality": "audio",
        "mime_type": "audio/mpeg",
        "uri": "https://example.com/recording.mp3"
      },
      {
        "type": "uri",
        "modality": "video",
        "mime_type": "video/mp4",
        "uri": "https://example.com/demo.mp4"
      },
      {
        "type": "uri",
        "modality": "document",
        "mime_type": "application/pdf",
        "uri": "https://example.com/report.pdf"
      }
    ]
  }
]

Choosing Among the Three Methods

If media files come from agent runtime attachments, prefer TLS plugin/SDK for parsing, upload, and reference reporting.

If customers have existing TOS private resources, prefer object-reference association for observability linking.

If media already has publicly accessible URLs, use GenAI standard multimedia messages for display.

All three target different data sources but share the same goal: return images, audio, video, and other multimodal content to concrete messages and call nodes.

Quick Verification Steps

Start with a simple, repeatable image-text request to validate the pipeline before expanding to complex workflows.

Configure collection : Install the AgentLoop collection plugin supporting the target media scenario, set TLS region, collection target, and auth, and enable content collection as needed.

Initiate request : Send a task containing an image. After completion, verify media indicators, image preview, prompt, and related call records in Session and Trace.

Check anomalies : If images don't appear, combine plugin logs to inspect attachment reading, upload authorization, and file upload results, then confirm access permissions and network conditions.

Images, prompts, and tool results may contain business data. Before production integration, define allowed collection scope, accessible personnel, retention periods, and coordinate trace and media file retention policies.

From Seeing the Process to Understanding the Context

As agent inputs expand beyond text, TLS AgentLoop's multimodal solution places media back into specific execution contexts: developers can view task images and simultaneously compare prompts, model outputs, and tool results to understand the processing flow.

For developers, this adds a directly inspectable content basis during debugging. For teams, it centers problem discussion on a single session and call chain.

On top of AgentLoop's existing data replay and evaluation capabilities, these image-text task records can seed problem case organization and sample construction. Future multimodal sample replay and evaluation can be integrated per concrete data formats and evaluator support.

The endpoint of multimodal observability is not merely showing one more image, but letting developers see not only "what executed" but also "what content those executions were based on."