F²Agent: Multimodal Agent Fusion Paradigm Outperforms 16 Trading Baselines

Abstract

As financial information sources become increasingly diverse and heterogeneous, effective use of multimodal data is critical for high-quality trading. While LLM-based agents can accept multimodal inputs, existing methods suffer from three defects: limited multimodal modeling capability, inefficient fusion mechanisms, and insufficient robustness.

This paper proposes F²Agent (Financial Fusion of Agentic Intelligence) — a novel multimodal trading paradigm driven by financial fusion agent intelligence. F²Agent first deploys hierarchical specialized agents (market analysis, technical analysis, news analysis, sentiment analysis) to comprehensively extract modality-specific signals; then introduces a modality-aware adaptive fusion mechanism to dynamically capture fine-grained cross-modal dependencies; and finally applies noise-robust consistency regularization to generate noise-resistant trading signals. Experiments on six stocks and cryptocurrency assets show F²Agent consistently surpasses 16 competitive baselines across multiple trading metrics, with average relative improvement in annualized return exceeding 20%, reaching 120.48% on GOOG and 148.41% on TSLA. Paper link: https://arxiv.org/pdf/2608.05668

Background

Financial markets are driven by complex interactions of heterogeneous information sources — from quantitative historical prices to qualitative textual narratives. The efficient market hypothesis posits that prices reflect available information, yet structural differences across modalities and high noise in financial streams make digesting these multimodal signals difficult.

Evolution of Existing Methods

Early systems relied on rules and expert indicators (MACD, SMA), followed by ML/DL methods (LSTM, Transformer) learning predictive patterns from statistical and neural architectures, then reinforcement learning frameworks (DQN, PPO) optimizing sequential trading strategies. Recently, LLM-driven trading systems emerged: FinGPT transforms financial news and price information into trading decisions, FinAgent executes market reasoning with financial-domain LLM agents, TradingAgents enables multi-agent collaborative decision-making, and DeepFund provides end-to-end fund management.

Common Defects of Current LLM Trading Systems

These systems emphasize role-playing and task coordination but lack principled mechanisms for fusing heterogeneous modalities. In practice they rely on prompt-level concatenation — stuffing numerical time series as text tokens into the context window. This ad-hoc fusion fails to bridge the modality gap: LLM's text bias dominates joint representations, drowning subtle quantitative cues, leading to weak cross-modal interaction modeling and decisions vulnerable to market noise.

Problem Formulation

F²Agent addresses multimodal stock trend prediction. Given a stock set S and time range T, the multimodal input for stock s on date t comprises features from each modality (market, technical, news, sentiment). With K agents each producing a hidden representation, the fusion module concatenates agent features and maps them via a prediction head to category probabilities over {UP, DOWN}: Pr(y_t_s ∈ L | X_t_s) = F_θ(f(X_t_s)) Ground-truth labels are defined by adjusted close price p_t_c: Y_t_s = 0 (price down) or Y_t_s = 1 (price up).

The paper identifies three key challenges for reliable multimodal LLM trading agents:

Limited Multimodal Modeling : Financial signals originate from heterogeneous sources (numerical market data, technical indicators, news articles, social media, analyst reports) that jointly shape market dynamics. Existing methods either focus on price/technical indicators or rely on textual sentiment, failing to exploit cross-modal complementary information. For example, in the classic "sell the news" scenario, positive news sentiment may accompany bearish price action — isolated modality processing cannot detect cross-source misalignment.

Inefficient Fusion Mechanisms : Even when multiple modalities are fed to LLM agents, aligning heterogeneous signals with different structures, semantics, and temporal granularities remains difficult. Prompt-level concatenation treats numerical time series as text tokens; LLM's text bias dominates joint representations, drowning subtle quantitative cues and yielding weak cross-modal interaction modeling.

Insufficient Robustness : Financial streams are inherently stochastic with low signal-to-noise ratios and frequent market regime switches. Existing agents lack explicit mechanisms to separate persistent market trends from transient perturbations, causing overreactions to irrelevant news cues or minor price fluctuations.

Method

4.1 Hierarchical Specialized Agents

F²Agent deploys four specialized agents, each handling a distinct modality:

Market Analysis Agent (MA) : Encodes raw OHLCV (Open, High, Low, Close, Volume) sequences into high-level latent representations using a causal Transformer encoder to capture long-range temporal dependencies and lead-lag effects within a lookback window. After positional encoding and L_MA causal Transformer blocks, the [CLS] representation is taken as fixed-length summary h_s,t_MA.

Technical Analysis Agent (TA) : Focuses on expression-based fundamental factors (e.g., MACD, RSI) providing structured quantitative descriptions of trend, momentum, and volatility. Also uses a causal Transformer encoder to model technical indicator sequences, outputting technical representation H_s,t_TA.

Sentiment Analysis Agent (SA) : Uses DeepSeek-R1 (Distill-Llama-8B) to capture sentiment signals from financial news and social media. Supports tool-augmented retrieval: given (s, t), queries external news APIs (Alpaca, Alpha Vantage), retrieves articles, deduplicates and summarizes them into a news summary set. For each summary, predicts a binary sentiment label and brief rationale, extracts last-layer hidden states via last-token pooling, and averages across entries to obtain sentiment representation h_s,t_SA.

News Analysis Agent (NA) : Uses a two-stage fine-tuned Qwen2.5-7B-Instruct model with Chain-of-Thought prompting to generate transparent explanations. Given (s, t) and the news summary set, outputs a directional signal (UP/DOWN), confidence score, and natural language explanation. Fine-tuning uses a joint objective comprising language modeling loss and auxiliary classification cross-entropy. To mitigate majority-class bias, the prompt explicitly requires choosing from {UP, DOWN}.

4.2 Modality-Aware Adaptive Fusion Mechanism

Inspired by multimodal LLMs and MoE, F²Agent designs a modality-aware fusion layer as a coordination layer.

Modality Embedding Extraction and Alignment

Each modality sequence is processed by its specialized agent into a sequence representation. Fixed-length summaries are extracted (last-token pooling for LLM agents, [CLS] for Transformer encoders) and projected via learnable projection P_m into a shared d-dimensional space:

e_m = P_m · φ_m(f_m(X^(m))) ∈ ℝ^d,  P_m ∈ ℝ^(d×d̃_m)

Adaptive Modality Attention

Aligned embeddings compute modality-specific queries Q_m, keys K_m, values V_m, concatenated into joint query Q_all. Softmax attention yields fused representation R_final = Σ α_m V_m.

Modality Prior Injection

Learnable modality vectors r_m = u_m · A are injected into each modality's keys and values, preserving instance-agnostic evidence while introducing modality-specific biases. To encourage modality diversity and avoid trivial scaling, a regularization term is imposed:

L_modality = Σ_{m≠n} (r_m^⊤ r_n / (‖r_m‖·‖r_n‖))² + Σ_m ‖r_m‖²

4.3 Noise-Robust Consistency Regularization

Financial signals are noisy; some modalities may be missing or corrupted. To enhance robustness, a modality m* is randomly selected and its embedding perturbed with Gaussian noise, yielding perturbed input x'. Let s(x) and s(x') be prediction logits for original and perturbed inputs. Two constraints are applied:

L_robust = ‖s(x) − s(x')‖ + γ · max(0, α'_m* − α_m*)

The first term enforces logits-level consistency ; the second term suppresses over-reliance on the perturbed modality . Total loss:

L = L_CE + λ_mod · L_modality + λ_rob · L_robust

4.4 End-to-End Multimodal Agent Trading System

F²Agent provides a complete end-to-end trading system: a News Summarizer retrieves relevant articles via semantic vector search, ranks by market impact, and compresses into concise daily briefs; a Modality Processing Module encodes the four modalities with specialized agents; a Backtesting Engine converts prediction signals into executable actions on historical data per trading strategy, evaluating profit and risk metrics. The overall pipeline integrates modality-specialized agents, fusion module, and trading output into a unified system.

Experiments

Experimental Setup

Datasets : Six assets — AAPL, AMZN, GOOG, MSFT, TSLA (five stocks) and BTCUSD (one cryptocurrency).

Evaluation Metrics : Annualized Return Rate (ARR), Cumulative Return (CR), Sharpe Ratio (SR), Maximum Drawdown (MDD).

Baselines : 16 baselines across four categories — market/rule strategies (B&H, MACD, ZMR, SMA), deep learning (LSTM, Transformer), reinforcement learning (DQN, PPO), general LLMs (Qwen3-8B, DeepSeek-R1-0528, Llama4-Scout-17B, GPT-5-mini), and financial LLM agents (FinGPT, FinAgent, TradingAgents, DeepFund). All experiments run on 6 NVIDIA A40 GPUs.

Main Results

F²Agent achieves average ARR rank 1.00 (best on all six assets) , and SR best on five assets. Detailed performance:

AAPL : ARR 50.08%, SR 1.22, MDD 7.83%, vs. strongest baseline DeepFund (ARR 33.83%) +48.03% .

AMZN : ARR 40.87%, SR 1.18, MDD 5.57%, vs. FinAgent (ARR 24.76%) +40.79% .

GOOG : ARR 120.48%, SR 2.55, MDD 7.01%, vs. FinAgent (ARR 108.16%) +11.39%, SR +18.60%.

MSFT : ARR 84.14%, SR 2.08, MDD 8.03%.

TSLA : ARR 148.41%, SR 1.94, MDD 12.60%.

BTCUSD : ARR 53.57%, SR 1.52, MDD 9.79%.

Closest competitor FinAgent averages rank 3.17; market benchmark B&H ranks 3.50. F²Agent achieves up to 48.03% relative ARR improvement and 18.60% SR improvement, with average relative improvement exceeding 20% .

Ablation Studies

Ablation validates each fusion component's contribution, using AAPL and BTCUSD as representatives:

Concatenation Fusion vs. F²Agent : AAPL ARR jumps from 19.21% to 50.08%; BTCUSD from 7.18% to 53.57%, showing simple feature concatenation is far less effective than modality-aware fusion.

Remove Modality Prior : AAPL ARR drops to 38.04%, indicating modality priors guide the fusion process.

Remove Robust Regularization : AAPL ARR drops to 34.77%, showing robust regularization improves stability under noise.

Remove Both : Largest drop (AAPL to 27.28%), confirming the two components are complementary.

Robustness Studies

Tested on AAPL and BTCUSD with varying modality combinations. Baseline methods often degrade as modalities increase — e.g., Llama4-Scout-17B on AAPL drops after adding technical indicators; FinAgent on BTCUSD sees lower risk-adjusted returns after multimodal integration. In contrast, F²Agent remains stable and benefits from added modalities , achieving strongest or near-strongest risk-adjusted performance in the full-modality setting.

Token Efficiency and Case Studies

F²Agent consumes fewer tokens than competing frameworks like TradingAgents and FinAgent. This efficiency stems from architectural design: computation is distributed to specialized agents; token-intensive reasoning is confined to news and sentiment agents (where text understanding is essential), while other components rely on structured processing to reduce overhead.

Qualitative analysis on AAPL (Apr–Oct 2025) shows F²Agent better handles noisy narratives and cross-modal reasoning compared to strongest baseline DeepFund. For instance, on Sep 22, DeepFund issued BUY based on exaggerated iPhone 17 pre-order narrative, while F²Agent detected insufficient cross-modal support and chose HOLD; on May 22, DeepFund issued BUY due to oversold technical signal, but F²Agent chose SELL based on news-driven risk evidence. This demonstrates F²Agent relies on cross-modal evidence rather than isolated modality cues for decisions .