A First Systematic Review of Multimodal Agentic Frameworks

Multimodal agents are defined as frameworks centered on a large multimodal model (LMM) that perceive, reason, remember, plan, and act across multiple modalities within a closed cognitive loop.

Four Main Contributions

Modal‑based taxonomy : analyzes how multimodality reshapes the architecture of each functional module (Perception, Reasoning, Planning, Memory, Action).

Three‑category perception‑fusion strategy : Delegated → Late‑fusion → Early‑fusion, with empirical evidence that multimodal‑fusion frameworks generally outperform language‑only intermediate representations.

Cross‑domain performance evaluation : links architectural choices to quantitative gains in four application domains (pure‑text, robotics, GUI/web navigation, multimedia generation).

Efficiency‑scalability trade‑off map : compares resource usage, latency, and deployment cost across fusion strategies, noting that fine‑tuned domain‑specific architectures are more economical and lower‑latency than proprietary token‑intensive APIs.

Three Pillars of Agent Frameworks

Orchestrator (Prefrontal Cortex)

The orchestrator is the central decision unit responsible for reasoning, planning, and memory.

Reasoning : chain‑of‑thought (CoT) → zero‑shot “Let’s think step by step” → self‑consistent multi‑path sampling.

Planning : decomposes tasks into subtasks with a “plan‑observe‑revise” loop; advanced approaches encode plans as executable code (Code as Policies) or train models to call APIs directly (ToolFormer).

Memory : three‑tier structure – working memory (extended context window), episodic memory (store and reflect on past experiences, e.g., Reflexion), semantic memory (structured knowledge retrieval, e.g., RAG).

Perception (Source of Multimodal Capability)

Perception evolves through three stages to mitigate information loss.

Action (Closing the Perception‑Action Loop)

Action materializes plans into executable operations: tool/API calls (ToolFormer, Gorilla), code execution, Web/UI navigation (WebVoyager, AppAgent), and robot control (RT‑2, OpenVLA). Results are written back to memory.

Modalities

Text : foundational modality for agents such as WebGPT, ReAct, MetaGPT.

Image : used for tool orchestration (VisProg, MM‑ReAct) and direct action output (RT‑2, SeeAct).

Video : introduces temporal dependencies and high computational cost (VideoAgent, LLoVi, VideoMind).

Audio : focuses on fine‑grained reasoning and generation (WavCraft, SonicRAG, ReelWave).

How Multimodality Reshapes Each Module

Three Perception Strategies

Delegated Perception : LLM acts as scheduler, delegating vision/audio tasks to expert models that return captions, detection boxes, or transcriptions (e.g., HuggingGPT, MM‑ReAct, Chameleon).

Late‑Fusion Perception : freezes visual encoder (CLIP/ViT) and adds a trainable projection layer; Flamingo’s Perceiver Resampler is a prototype, spawning embodied controllers (RT‑2, OpenVLA) and GUI agents (CogAgent, Ferret‑UI).

Early‑Fusion Perception : tokenizes all modalities and shares self‑attention layers for native cross‑modal attention (e.g., GPT‑4o, Gemini 1.5, Chameleon‑FAIR, Fuyu‑8B, Transfusion, Show‑o, Janus‑Pro).

Three Reasoning & Planning Strategies

Language‑based reasoning : reasoning occurs entirely in language space; images are described before reasoning (CoT, ToT, ReAct, Reflexion).

Visual grounding reasoning : reasoning directly references visual representations—spatial coordinates, set‑of‑marks (SeeAct, WebVoyager, Magma, RT‑2).

Cross‑modal reasoning : integrates asynchronous heterogeneous sensory inputs (ImageBind six‑modality space, NExT‑GPT any‑to‑any, CoDi‑2, ReelWave).

Memory Architecture: Modality‑Specific vs Unified

Modality‑specific memory : stores different data types separately, bridging via text indices or symbolic pointers (HuggingGPT’s JSON history, VideoAgent’s “video as database”).

Unified memory : projects all modalities into a shared semantic embedding space; working memory natively supports interleaved image‑text‑audio‑video sequences (Gemini 1.5, GPT‑4o); episodic memory enables cross‑modal retrieval (HM‑RAG); semantic memory organizes by “concept” rather than format (ImageBind, SonicRAG).

Temporal context management : approaches such as Mobile‑Agent‑v2’s rolling summary, Reflexion’s distillation, AppAgent‑v2’s RAG, GUI‑Owl’s limited recent screenshots illustrate inevitable compression loss, which is amplified in multimodal scenarios.

Three Expansions of Action Space

Language‑driven action : structured text commands (JSON, function calls) exemplified by ToolFormer, Gorilla, ToolLLM (over 16,000 real APIs).

Visual‑grounded action : pixel‑level coordinate interaction (AppAgent clicks/swipes, CogAgent high‑resolution GUI perception, VoxPoser 3D value maps).

Embodied multimodal action : continuous motor‑control signals; VLA models tokenize actions generated by base models (RT‑2, OpenVLA, SayCan, PaLM‑E).

Evaluation Methods

Five categories are used: environment task completion, programmatic functional correctness, deterministic automatic metrics, human evaluation, and LLM‑as‑a‑Judge. LLM‑as‑a‑Judge accounts for ~5% of evaluations and is limited to GUI/web navigation (e.g., WebVoyager uses GPT‑4V for trajectory scoring, WebArena uses GPT‑4 fuzzy matching). Robotics and long‑video domains rely solely on deterministic metrics.

Four Application Domains

Pure‑Text Applications

Q&A and self‑correction : progression from WebGPT (web browsing + citation) → ReAct (reason‑action interleaving) → Reflexion (language‑based self‑reflection RL, SOTA on ALFWorld and HotpotQA).

Multi‑agent collaboration & software development : Generative Agents (social simulation of 25 agents) → MetaGPT/ChatDev (simulated software company division of labor).

Robotics & Physical Embodiment

Three‑stage evolution:

MLLM multimodal reasoning : ViLa (GPT‑4V directly participates in perception‑reasoning), GPTArm (hierarchical processing), COME‑robot (dual‑loop system), EMMA (autonomous driving with direct camera‑to‑trajectory mapping).

VLA models (current frontier) : PaLM‑E → Magma → RT‑2 → Octo → OpenVLA → π₀/π₀.₅ (flow‑matched high‑frequency continuous actions) → VTAM (tactile) → GR00T N1 (humanoid robot) → CoT‑VLA (visual chain‑of‑thought, 17% improvement) → HALO (embodied multimodal CoT).

GUI & Web Navigation

Digital environments present a massive, dynamic action space (thousands of interactive elements).

Fine‑tuned late‑fusion agents : WebGUM, Pix2Act, CogAgent (1120×1120 high‑resolution encoder), Ferret‑UI, SeeClick (direct coordinate prediction), GUI‑Owl.

Native multimodal agents : SeeAct, WebVoyager, AppAgent, Mobile‑Agent (self‑reflection), MM‑Navigator, UI‑TARS‑2 (multi‑round RL, OSWorld 47.5%), Agent S2, OpenCUA (22,600‑task open dataset, OSWorld‑Verified 45.0%).

Multimedia Content Generation & Editing

Native multimodal : GenArtist (GPT‑4o planning tree + visual self‑correction), CoSTA_/FaSTA_ (A* search tool path + fast/slow planning), FLATTEN/UniEdit/AnyV2V (zero‑shot video editing).

Multi‑agent : CREA (creative director / art critic / optimization strategist), ReelWave (film‑audio production team), EditDuet (editing + commentary agents), Paper2Poster/Paper2Video (paper‑to‑poster/video).

Long‑Video Understanding & Retrieval

Long video generates hundreds to thousands of visual tokens per minute, causing quadratic attention cost. Agents treat video as an explorable, queryable environment rather than raw input.

Specialized fine‑tuned reasoning agents : VideoMind (Chain‑of‑LoRA with dynamic planner/grounder/verifier roles, 2B → 78B model) and VLog (video narration as lexical units for generative retrieval).

Native multimodal agents : Gemini 1.5 Pro / GPT‑4o ingest hour‑long video directly, but long‑range causal reasoning still requires external agent structures (DoraemonGPT, VideoRAG, VideoDeepResearch, DVD – LVBench 74.2%).

A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applications
https://arxiv.org/abs/2608.20379v1