Paper Reading: ArchetypeTrader – A Reinforcement‑Learning Framework for Selecting and Optimizing Crypto Trading Strategies

Background

Quantitative trading uses mathematical models and automated execution. Reinforcement learning (RL) has shown promise for developing profitable strategies in the highly volatile cryptocurrency market, but existing crypto‑RL methods suffer from two key flaws: (1) handcrafted market segmentation (e.g., bull/bear labels) oversimplifies market dynamics and hides subtle profitable opportunities; (2) inefficient use of demonstration data introduces noise and bias, leading to volatile returns and severe drawdowns.

Problem Definition

The paper isolates the market‑segmentation problem and the demonstration‑data utilization problem as the primary obstacles to robust crypto‑RL trading.

Method

3.1 Archetype Discovery

Dynamic programming (DP) samples fixed‑length ( h) data blocks from the training set and generates an optimal trading action for each block, limiting each block to a single trade to reduce noise. An LSTM‑based encoder q_{\theta_e}(z_e\mid s_{demo}, a_{demo}, r_{demo}) maps each demonstration trajectory to a continuous embedding z_e. A vector‑quantization (VQ) module discretizes z_e to a code z_q selected from a codebook of size K. A decoder p_{\theta_d}(\hat{a}_{demo}\mid s_{demo}, z_q) reconstructs the original action a_{demo}. The combined loss L_{rec} measures reconstruction error, while additional terms keep the selected code close to the encoder output.

3.2 Archetype Selection

The basic Markov Decision Process (MDP) is lifted to a hierarchical (level‑ H) MDP. For a fixed horizon H=[t, t+h-1], the state s_{sel} is the market observation at the start of the horizon, and the action a_{sel} selects an archetype from the learned codebook. The reward for selecting an archetype is the sum of step‑wise rewards over the entire horizon.

The policy \pi_{\phi} is optimized to maximize expected return while encouraging consistency with demonstration trajectories. The objective includes an environment‑reward term and a KL‑divergence penalty weighted by \alpha.

3.3 Archetype Refinement

The refinement stage formulates a step‑level MDP. The agent observes a real‑time state s_{ref}^{\tau} composed of market observation s_{ref1}^{\tau} and archetype information s_{ref2}^{\tau}. A strategy adapter generates an optimization signal a_{ref}^{\tau} that adjusts the base archetype action a_{base}^{\tau} without overwriting it, producing the final action a_{final}^{\tau}. A regret‑aware reward computes the top‑5 hindsight optimal adjustments; the cumulative return R, baseline return R_{base}, and optimal DP return R_{opt}^1 are combined with hyper‑parameters \beta_1 and \beta_2 to guide learning.

The training objective minimizes cross‑entropy loss L between the adapter’s output and the optimal adjustment a_{ref}^{\tau}.

Experiments

4.1 Datasets

Four cryptocurrency pairs (BTC/USDT, ETH/USDT, DOT/USDT, BNB/USDT) with 10‑minute bars and order‑book depth M=25 are used. Data are split into training (2021‑06‑01 to 2023‑05‑31), validation (2023‑06‑01 to 2023‑12‑31), and testing (2024‑01‑01 to 2024‑09‑01).

4.2 Evaluation Metrics

Total Return (TR), annualized volatility, maximum drawdown (MDD), annual Sharpe ratio (ASR), annual Calmar ratio (ACR), and annual Sortino ratio (ASoR) are reported.

4.3 Baselines

Eight baselines are compared: standard RL methods (DQN, PPO, CDQNRP, CLSTM‑PPO), hierarchical RL methods (EarnHFT, MacroHFT), and rule‑based strategies (IV, MACD).

4.4 Experimental Settings

Experiments run on four RTX‑4090 GPUs; transaction cost \delta = 0.02\%; maximum position limits differ per asset (BTC 8, ETH 100, DOT 2500, BNB 200).

Archetype discovery samples 30 k DP trajectories with segment length h=72, trains a VQ encoder‑decoder (128 hidden units, embedding dimension 16, codebook size K=10, \beta_0=0.25) for 100 epochs.

Archetype selection optimizes the RL selector for 3 M steps, keeping the checkpoint with best validation performance; \alpha=1.

Archetype refinement trains the regret‑aware adapter for 1 M steps, with \beta_2=1 and asset‑specific \beta_1 in {0.3, 0.5, 0.7} (BTC/DOT 0.5, ETH/BNB 0.7).

4.5 Results and Analysis

Overall Performance

ArchetypeTrader achieves the highest profit and best risk‑adjusted returns in all markets except BNB/USDT, where it remains competitive despite a largely bullish market. Single‑policy RL methods struggle with regime changes; rule‑based methods perform well on specific datasets but suffer catastrophic losses on others; hierarchical RL methods are more stable but still limited by handcrafted indicators.

Archetype Interpretability

Visualization of selected archetype signals on BTC/USDT shows distinct behaviors: archetype 9 is suited for bear periods with a “short‑and‑hold” style, while archetype 2 captures short‑term reversals using a contrarian or mean‑reversion approach.

Ablation Studies

Embedding ablation compares continuous embeddings, clustered embeddings, and discrete VQ embeddings; the VQ embedding consistently yields higher returns and better risk metrics. Adding the strategy adapter captures additional trading opportunities and corrects sub‑optimal archetype execution, improving profit and risk control. Removing the regret‑aware penalty causes the refined policy to underperform the baseline selector.

Hyper‑parameter Sensitivity

On the DOT/USDT dataset, varying the number of archetypes K and segment length h shows that intermediate values balance expressive power and agility, achieving the best performance.