Foundations and Frontiers of Multimodal Agentic Frameworks: A Comprehensive Survey

Introduction

Large language models have expanded agent capabilities from tool‑calling to perception, memory, planning, and action. Text‑only agents face a bottleneck because real‑world information is multimodal (images, video, audio, sensor data). Multimodal large models enable agents to ingest these signals directly, reshaping the perception‑memory‑action loop.

Framework Foundations

The authors abstract a multimodal agent into three tightly coupled modules: Perception (converts multimodal inputs into decision‑ready representations), Orchestrator (handles reasoning, planning, and memory), and Action (executes tool calls, UI interactions, or embodied control). This structure guides the subsequent analysis.

Taxonomy of Multimodal Perception

Three perception strategies are identified:

Delegated perception : external tools translate non‑text modalities into natural language for the language model; low engineering cost but prone to information loss and tool‑quality dependence.

Late‑fusion perception : modality‑specific encoders map inputs to representations consumable by the language model; retains more features but requires task‑specific training.

Early‑fusion perception : a unified multimodal model processes raw inputs in a shared embedding space; offers end‑to‑end understanding at the expense of higher compute, memory, and context management.

Core Components

Orchestrator

The orchestrator acts as the execution core, integrating reasoning (interpreting observations and task constraints), planning (decomposing goals into steps and adapting plans), and memory (storing short‑term context, long‑term experience, and reusable knowledge). Multimodal tasks increase input complexity, requiring the orchestrator to assess modality reliability and historical influence.

Perception & Action

Perception modules evolve from tool‑based pipelines (image captioning, object detection, speech‑to‑text) to unified multimodal models that embed text, images, audio, and video together. Action modules expand from pure text output to UI clicks, drag‑and‑drop, robot motion, and multimedia generation, forming a closed perception‑action loop.

Memory Architectures

Three memory designs are discussed:

Modality‑specific memory stores separate streams (text, image, video, sensor) for efficient retrieval but incurs high integration cost.

Unified memory projects all modalities into a common space, facilitating cross‑modal retrieval and planning.

Temporal context management handles long‑range video or multi‑turn interactions, preventing loss of early events.

Bandwidth management is critical: agents must decide which visual/audio data to retain (raw frames, features, summaries, or indexes) to balance fidelity, retrieval speed, and inference cost.

Applications

The survey covers four representative domains:

Robotics & embodied agents : translate language goals into physical actions, requiring continuous perception, error handling, and safety constraints.

Interface & web navigation : understand screenshots, buttons, and layouts to perform clicks, inputs, and scrolling; vulnerable to layout changes and visual ambiguity.

Multimedia generation & editing : multi‑step pipelines that plan content structure, invoke generative models, verify results, and iteratively refine style.

Long‑video understanding & retrieval : locate events across thousands of frames, requiring key‑frame extraction, temporal indexing, and hierarchical summarization.

Performance Analysis

Performance bottlenecks differ by domain: robotics emphasizes task‑completion and safety, web navigation stresses element localization, long‑video tasks focus on event recall and temporal precision, while multimedia generation values aesthetic consistency and controllability. No single “universal score” exists; evaluation must align with task‑specific metrics.

Efficiency

Multimodal agents incur higher costs than text‑only agents due to encoding images/audio/video and tool‑call latency. Delegated perception offloads encoding but adds communication overhead; late‑fusion reuses modality encoders; early‑fusion demands more GPU memory and token budget. System‑level optimizations include separating high‑level planning from low‑level control, caching page structures, batching generation calls, and selective key‑frame processing.

Scalability

Scalability concerns task length, environment complexity, and modality diversity. Expanding from single tasks to open environments introduces distribution shift in modalities, tool‑interface changes, and evolving safety boundaries. Modular perception, interchangeable tools, stable memory, and auditable execution traces are essential for scalable designs.

Latency

Latency sources: multimodal encoding, main model inference, external tool calls, and environment feedback. Delegated architectures suffer from tool‑call delays; late‑fusion from encoder latency; early‑fusion from large context windows. Effective systems use hierarchical processing, invoking heavy models only when uncertainty is high.

Safety and Robustness

Risks extend beyond textual hallucination to visual misrecognition, missed video events, robot mis‑predictions, and unsafe generated content. Governance mechanisms must span perception (noise handling, adversarial robustness), planning (uncertainty detection, human confirmation), and action (risk‑limited operations, rollback, audit logs).

Limitations

Current research is fragmented: systems are evaluated on narrow tasks with heterogeneous datasets, models, tools, and metrics, making cross‑paper comparison difficult. Deploying to real environments faces high multimodal input costs, uncontrolled state changes, lack of explainability, and insufficient safety‑by‑design.

Future Directions

Unified cross‑modal representations with controllable uncertainty.

Long‑term memory that stores visual evidence, action trajectories, and user preferences, supporting continual learning without breaking safety guarantees.

Real‑world benchmarking that records full execution traces, tool calls, error recovery, and human interventions across robotics, web interaction, and multimedia creation.

Robust governance frameworks integrating permission control, risk grading, reversible actions, audit logging, and human‑in‑the‑loop oversight.

Conclusion

The survey organizes multimodal agent research into a clear technical map: perception‑orchestrator‑action architecture, three perception paradigms, four application pillars, and a multi‑dimensional evaluation of performance, efficiency, scalability, latency, and safety. Reliable multimodal agents must jointly solve perception grounding, cross‑modal reasoning, long‑term memory, executable actions, process evaluation, and safety governance, shifting the competitive focus from model size to system integration.

Code example

来源：专知
本文
约6000字
，建议阅读
10
分钟
这篇综述系统梳理多模态智能体框架的基础模块、技术谱系和应用前沿。