FAST: A Parallel Framework that Accelerates Reinforcement Learning for Autonomous Driving

Background: Why autonomous‑driving RL training is slow

Reinforcement‑learning (RL) for autonomous driving relies on massive interaction data from simulation environments, making sampling efficiency the key determinant of training speed. Existing approaches use synchronous parallelism to increase data throughput, but in closed‑loop RL the duration of each driving scenario varies dramatically: rare, safety‑critical scenes terminate quickly, while normal scenes continue for long periods. This heterogeneity forces a global reset after any episode ends (Synchronous Global Reset, SGR), incurring costly re‑loading of high‑definition maps, traffic flow reconstruction, and sensor state initialization. As parallel scale grows, the probability of a short episode triggering a reset approaches 1, causing reset overhead to dominate the sampling cycle, which leads to (1) reduced sampling efficiency and (2) degraded data quality because long‑term trajectories are frequently truncated.

Sampling efficiency regression: larger parallel scale results in more frequent resets, increasing total sampling time.

Data‑quality degradation: frequent truncation prevents the agent from learning long‑tail behaviors and complex decision chains.

Problem Definition: What determines effective throughput?

The authors define effective throughput as the number of valid on‑policy transitions collected per unit time. It is decomposed into three inter‑dependent factors:

Raw sampling throughput – total transitions generated per unit time, reflecting overall system output. It is reduced when reset waiting time grows.

Sample efficiency – proportion of generated transitions that are valid for learning, reflecting data quality. Traditional SGR guarantees 100% sample efficiency but at the cost of time.

Time‑efficiency ratio – fraction of actual simulation time within the total sampling cycle, indicating how much reset overhead drags down throughput.

The three factors conflict in SGR: while it keeps sample efficiency high, both raw throughput and time‑efficiency ratio collapse as parallelism increases.

FAST Framework: Decoupling “individual termination” from “global reset”

FAST’s core insight is to break the coupling between an environment’s termination and the global reset. Terminated environments continue with dummy (virtual) steps to keep tensor shapes consistent, while still‑running environments keep collecting useful trajectories until a unified truncation condition is met.

1. Dynamic Parallel Sampling Alignment (DPSA)

When an environment finishes, it no longer triggers an immediate reset. Instead, it produces dummy data that is masked as invalid for gradient computation. At each timestep the system computes a global termination rate – the proportion of terminated clips among all clips. If this rate exceeds a predefined threshold or a maximum time horizon is reached, a unified truncation is performed. DPSA yields two key effects: (a) surviving environments retain long‑term trajectories, and (b) global reset frequency drops dramatically, keeping the time‑efficiency ratio above 99.6%.

2. Scaled Mask‑Padding Optimization (SMPO)

Virtual steps introduce padding transitions that would corrupt advantage‑function estimates if left untreated. Moreover, the density of padding samples changes with training progress, destabilising update magnitudes. SMPO maintains a per‑timestep validity mask and normalises loss by the total number of valid transitions. The optimisation objective incorporates the probability ratio between new and old policies, the estimated advantage, and a clipping hyper‑parameter. By normalising with the count of valid transitions, SMPO guarantees that padding data has no impact on policy updates, preserving an unbiased on‑policy distribution.

Experimental Results: Faster, more stable, zero‑loss performance

1. Execution time reduction

On an 8‑GPU node (NVIDIA H20) using DiDi’s real‑world takeover scenario dataset, FAST reduced sampling time for 10 parallel clips from 84.67 s (sequential baseline) to 14.35 s – an 83.0% reduction, yielding a 1.94× overall training speed‑up. FAST also outperformed SGR‑10 (1.42×) and VER‑10 (1.45×). At 30 clips, SGR’s speed‑up fell below 1× due to reset bottlenecks, while FAST maintained a >2× advantage.

2. Effective throughput and sample quality

FAST achieves a higher effective throughput because it simultaneously improves sample efficiency and time‑efficiency ratio. Early in training the sample efficiency is lower due to dummy steps, but it stabilises above 93% later, while the time‑efficiency ratio stays above 99.6% across all configurations. In contrast, SGR and VER keep 100% sample efficiency but suffer catastrophic drops in time‑efficiency.

3. Trajectory length and short‑time bias analysis

Traditional SGR shows a sharp decline in average episode length as parallel scale grows, caused by forced global resets that truncate long trajectories and introduce a short‑time bias. FAST maintains an average length of ~35 steps across all clip settings, matching the sequential baseline and eliminating the bias.

4. Overall training convergence speed

When plotted against wall‑clock training time, FAST‑10 reaches a stable performance plateau after roughly 20 hours, whereas the sequential baseline needs >40 hours and the accelerated baselines SGR‑10 and VER‑10 require ~30 hours.

5. Closed‑loop driving metrics validation

Jackknife resampling on 14 core closed‑loop driving metrics (collision, front‑car risk, safety margin, hard brake, hard acceleration, lane‑keeping, red‑light violation, yellow‑light entry, parking precision, lateral avoidance, speeding, efficiency, stall time, pull‑out deviation) shows that FAST‑10’s 95 % confidence intervals fully cover zero difference compared with the sequential baseline. This statistically confirms that FAST’s speed gains incur no degradation in safety, comfort, compliance, or efficiency.

Conclusion

FAST resolves the core bottleneck of parallel RL sampling for autonomous driving by combining DPSA’s dummy‑step mechanism with SMPO’s unbiased gradient handling. Experiments demonstrate that decoupling individual termination from global reset breaks the efficiency ceiling of synchronous parallel frameworks, delivering a plug‑and‑play high‑throughput solution applicable to any on‑policy RL setting with highly heterogeneous episode lengths, such as embodied intelligence and robotic manipulation.