Why Multimodal AI Agents Could Be the Next Killer App for Large Models

I first tested a multimodal AI agent on September 25, 2023, the day OpenAI released a multimodal ChatGPT. By setting a seafood restaurant in Newport Beach as the agent's hometown, I let it act as a newly hired Google engineer with a travel‑loving personality, feeding it my blog posts so it knew me better than most friends.

Multimodal Models

Modern multimodal LLMs such as LLaVA, Next‑GPT, MiniGPT‑4 and VisualGLM share a common structure: a large language model core, an encoder for images/audio/video, and a diffusion generator for output media. Training typically adds a projection layer between the encoder and the LLM and another between the LLM and the diffusion model, plus LoRA‑based instruction tuning.

Next‑GPT, for example, uses a 7B Vicuna backbone; the projection and LoRA layers add only 131 M parameters (≈1 % of the total), keeping GPU costs to a few hundred dollars. However, practical tests show poor image, audio and video quality, limited understanding of complex pictures, and sub‑par speech synthesis.

Two main approaches to image‑to‑text conversion are the CLIP Interrogator (CLIP + BLIP) and Dense Captions (CNN‑based). CLIP Interrogator captures style and relationships but is slower; Dense Captions is faster and more accurate for multi‑object scenes. Combining both can provide richer prompts for the LLM.

Memory

Current agents rely on Retrieval‑Augmented Generation (RAG) with TF‑IDF or vector databases, which struggles to recall rare personal facts (e.g., an old nickname) because the retrieval score is low and fine‑tuning data is scarce. A practical short‑term solution mixes RAG, fine‑tuning, and periodic text summarization of dialogue history.

Berkeley’s MemGPT integrates RAG and text‑summary modules, borrowing OS concepts like hierarchical storage and interrupts to extend context without modifying the base model.

Task Planning

Complex tasks such as multi‑hop QA or long‑document summarization require either very long context windows (100 K tokens) or sophisticated decomposition pipelines. AutoGPT attempts automatic decomposition but often fails on real‑world web pages due to Ajax loading and lack of visual parsing. Vision‑enabled models with higher input resolution could mitigate this.

Tool Creation

Plugins let LLMs invoke external services (e.g., DALL·E‑3). The system prompt for a DALL·E plugin is lengthy and includes safety policies; the article reproduces it inside a

block for reference.</p>
<pre><code>You are ChatGPT, a large language model trained by OpenAI, based on the GPT‑4 architecture.
Knowledge cutoff: 2022‑01
Current date: 2023‑10‑21
# Tools
## dalle
// ... (plugin policy omitted for brevity)</code>

Tool use suffers from hallucinations when the model decides to compute internally instead of calling the plugin, and from missing dynamic web content when parsing HTML directly.

Personality & Emotion

Agents can be modeled as digital twins (real‑world personas) or fantasy characters. Personality can be injected via fine‑tuning on character‑specific corpora or by quantifying traits with MBTI‑style vectors, as demonstrated by Paradot’s interface. Emotion is treated as a mutable state vector (e.g., “happiness”, “boredom”). Modern agents could store emotion as text or embeddings rather than raw vectors, but the underlying principle remains a continuously updated internal state.

Cost Challenges

Running agents 8 hours a day with GPT‑3.5 API quickly becomes financially unsustainable. Three avenues for cost reduction are:

Model routing: dispatch simple queries to small models and reserve large models for complex tasks.

Inference infra optimization: improve batch sizes, tensor‑core utilization, and KV‑cache reuse.

Hardware efficiency: leverage large‑memory GPUs (e.g., GH200) to store KV caches and reduce recomputation.

Development costs also remain high due to data collection, augmentation, fine‑tuning, vector DB construction and prompt engineering.

Evaluation of AI Agents

Evaluating agents is harder than evaluating pure LLMs because it involves interaction quality, long‑term coherence, and tool‑use correctness. Simple metrics like “number of dialogue rounds” are insufficient; human‑in‑the‑loop assessments, data‑augmentation tricks, and synthetic benchmarks are being explored.

Social & Reliability Issues

Legal and ethical questions arise around digital twins of public figures, copyrighted characters, and personal replicas. Reliability concerns include model hallucinations (e.g., fabricated awards) and system uptime—critical for both enterprise and consumer assistants.

Conclusion

While current multimodal agents demonstrate impressive capabilities, achieving low‑cost, high‑reliability, truly embodied AI with memory, personality, and autonomous tool use remains an open research frontier. The author believes that once these challenges are solved, AI agents will become the true "killer app" of large models.