How MetaTrader Uses Reinforcement Learning to Boost Trading Strategy Generalization

Background Reinforcement learning (RL) has shown strong potential for sequential portfolio optimization, but traditional offline approaches often overfit historical data and fail to generalize to the non‑stationary financial market. The authors frame the problem as a partially offline RL task and propose the MetaTrader method to address the "generalization‑optimality" dilemma.

Problem Definition

The portfolio optimization task is modeled as an MDP with an 8‑tuple (O, A, H, Z, P_h, P_z, R, γ). Observations O include daily open, close, high, low prices, volume, technical indicators, and covariance matrices. Actions A are continuous trade quantities per asset, later discretized for evaluation. The state is decoupled into market state h_t and balance state z_t, with separate transition dynamics P_h and P_z. Immediate reward R is the daily portfolio return, and γ controls the discount of future rewards.

Method

3.1 MetaTrader Overview To improve OOD generalization, the training set is split into M subsets, each further divided into sequences of length T=64. Training consists of two stages: OOD policy learning and in‑domain fine‑tuning. The core contribution is a novel TD learning strategy that aggregates transformed‑data TD targets to estimate worst‑case values.

3.2 OOD Data Transformations Three transformations simulate realistic market shifts:

F₁: Reverse the returns of the top α% assets with the highest price increase at each timestep, mimicking sudden short‑term shocks.

F₂: Invert the overall trend of a training subset to emulate long‑term market events.

F₃: Down‑sample the sequence by a factor Δ, compressing temporal dynamics to capture multi‑scale patterns.

3.3 Double‑Layer RL with Data Transform OOD policy learning samples a subset D(i) for inner‑loop optimization of agent and critic parameters, then performs an outer‑loop update using second‑order gradients across different data splits. The objective encourages robustness to OOD trajectories. In‑domain fine‑tuning uses recent data without transformations, applying the same double‑layer scheme but only on original data to adapt to recent market dynamics.

3.4 Worst‑Case Guided Transform‑Based TD Learning The critic loss incorporates a conservative TD target derived from transformed data. The inner‑loop TD target is defined as a worst‑case estimate over the batch of transformed samples, mitigating over‑optimistic value estimates common in limited offline datasets.

Experiments

4.1 Experimental Setup Two public datasets (CSI‑300 and NASDAQ‑100) from StockFormer are used. Data are split into training and test sets; the test set extends the original StockFormer test split. Baselines include market benchmarks, generic time‑series predictors, stock‑prediction models with a buy‑and‑hold rule, RL‑based trading methods, and other offline RL approaches. All methods account for transaction costs, and RL baselines average results over at least three random seeds.

4.2 Standard Offline Evaluation Training data from 2011‑01‑17 to 2018‑12‑31 are used for OOD learning; fine‑tuning occurs on the last year of training data (2018‑01‑04 to 2018‑12‑31). Testing covers 2019‑01‑01 onward (2019‑04‑01 to 2022‑04‑01). Metrics: cumulative return (CR), annualized return (AR), Sharpe ratio (SR), and maximum drawdown (MDD). MetaTrader outperforms all baselines, e.g., achieving 50% higher CR than FactorVAE on CSI and a 44.1% higher SR on NASDAQ; it also improves CR by 16.1% (CSI) and 32.7% (NASDAQ) over StockFormer.

4.3 Streaming Data Online Adaptation The test set is divided into three equal periods; each period is preceded by in‑domain fine‑tuning. MetaTrader shows significant gains over FactorVAE‑Finetune and StockFormer‑Finetune, with CR improvements of 26% (CSI) and >25% (NASDAQ) and SR lifts of ~18% and ~43% respectively.

4.4 Ablation Studies

Transform‑based TD integration: Adding the proposed transform‑based TD to a SAC baseline raises CR by 9.5% on CSI and outperforms other TD‑integration schemes.

In‑domain fine‑tuning: Double‑layer optimization yields +3.4% (CSI) and +12.1% (NASDAQ) CR; adding transformed data during fine‑tuning harms performance.

Impact of data transforms: Using all three transforms consistently improves all metrics; combining them yields an additional 10.8% CR gain in online adaptation on CSI.

4.5 Larger‑Scale Market Results Extending CSI to 587 stocks confirms MetaTrader’s scalability, maintaining superiority over market benchmarks and other RL‑based traders.

Conclusion

MetaTrader demonstrates that a double‑layer RL framework with OOD data augmentation and a worst‑case‑guided TD objective can effectively learn trading policies that generalize across distribution shifts, achieving state‑of‑the‑art performance on multiple financial benchmarks.