How an Interactive Imitation‑Learning Agent Framework Trains Robust Trading Strategies

Background

Reinforcement learning (RL) for real‑world algorithmic trading is hindered by a severe "simulation‑reality gap": static historical back‑testing ignores market impact, causing strategies that appear profitable in simulation to fail in live markets.

Problem Definition

The goal is to construct a more realistic training environment that overcomes static back‑testing limitations and the manual, rule‑based design of traditional agent‑based models.

Method

System Architecture

The proposed pipeline consists of three core components:

Data‑driven background agent pool : a set of imitation‑learning (IL) agents trained on different historical market regimes (e.g., bull, bear).

Action synthesis network : merges the actions of the IL agents into a responsive synthetic price trajectory.

Final RL trading agent : learns a robust trading policy within the interactive simulator.

Imitation‑Learning Agent Pool

Data split and feature engineering : Uses 2019‑2024 minute‑level OHLCV data of the Nasdaq‑100 ETF (QQQ). Significant local price extrema are identified to segment the data into distinct market regimes. Technical indicators (EMA, MACD, VWAP, relative volume) are concatenated into a state vector s_t and normalized with Min‑Max scaling.

Expert action labeling and IL training : For each regime segment, expert actions a_t (sell, hold, buy) are assigned based on whether the price moves more than 10%. Each segment trains a separate MLP with two hidden layers of 128 ReLU units, minimizing the mean‑squared error between the policy output π_{θ} and the expert label.

Interactive Market Simulator

The action‑synthesis MLP receives the action vectors from all IL agents. Its architecture includes an input layer matching the number of agents, a 10‑unit softmax layer that produces impact‑weight probabilities, hidden layers of 128 and 64 ReLU units, and a single output unit that predicts the next price change. The network is trained on retained historical segments to minimize MSE between the synthetic price trajectory and the actual price.

Final RL Trading Agent

MDP formulation : State s_t comprises the engineered technical indicators, current cash balance b_t, and stock holdings k_t. Action space A is continuous, representing the proportion of the portfolio to trade (negative = sell, positive = buy). Reward R is the change in portfolio value over a 230‑minute rolling window. Transition dynamics are governed by the simulator, which matches total buy and sell orders, applies a 0.5% transaction cost, and uses the action‑synthesis network to generate p_{t+1}, thereby embedding market impact.

RL algorithm: Deep Deterministic Policy Gradient (DDPG) : Maintains an actor network μ(s|θ_{μ}) and a critic network Q(s,a|θ_{Q}), each with target counterparts μ' and Q'. The critic loss minimizes (y_t - Q(s_t,a_t))^2 where y_t = r_t + γ Q'(s_{t+1}, μ'(s_{t+1})). The actor is updated via sampled policy gradient, and both target networks are softly updated with rate τ for stability.

Experiments

Setup

IL agents: MLPs with two 128‑unit hidden layers (ReLU). RL actor: two 128‑unit layers; critic: three 128‑unit layers (ReLU). Optimizer: Adam. Training performed on a server equipped with two GPUs (total 27 TFLOPS). Baselines: (1) long‑only buy‑and‑hold, (2) RL trained on the same data without the IL pool, and (3) an IL baseline trained on the same data.

Results

IL agent diversity : Evaluation of ten trained IL agents on unseen market segments shows a wide range of behaviors. For example, agent 3 learns a completely passive strategy (0 % turnover), agent 4 adopts a high‑turnover active style, and agent 2 achieves the best risk‑adjusted return, providing rich behavioral inputs for the simulator.

RL performance : On 12 out‑of‑sample periods, the RL agent attains higher average returns and significantly lower risk than all baselines. The average maximum drawdown drops from –9.11 % (long‑only and RL baseline) to –4.97 % for the proposed RL agent.

Case studies : In a severe bear market, the long‑only and RL baseline lose –23.2 %, while the RL agent reduces loss to –11.6 % by systematically decreasing positions and converting assets to cash. In a volatile sideways market, the RL agent’s loss is –2.8 % versus –5.7 % for baselines, achieved through frequent small trades that manage short‑term fluctuations.

Regime‑dependent analysis : The RL agent preserves capital effectively in bear markets, generates positive returns in sideways markets, and captures substantial gains in bull markets—outperforming the IL baseline across all regimes. Risk‑return scatter plots illustrate the RL strategy’s robustness and risk‑averse behavior.

Discussion

The work provides a technically sound, scalable solution to the simulation‑reality gap in financial AI.

Enhance simulator realism by integrating a limit‑order‑book (LOB) model and order‑matching engine to capture more complex market dynamics.

Extend the framework to multi‑asset portfolios, addressing cross‑asset impact and covariance‑aware allocation.

Validate the system through months of paper‑trading with a broker API before live deployment, identifying residual gaps between simulated and real market conditions.

Deploy governance tools—including an emergency stop switch, hard position limits, and real‑time monitoring of model drift and anomalous behavior—to ensure safe and reliable operation.

Overall, the proposed interactive environment demonstrates strong deployment potential for robust, real‑world algorithmic trading.