DriveTeach-VLA Bridges Autonomous Driving Scenes and Foundation Model Pre‑training via Image Trajectories

The paper, accepted at ECCV 2026, presents DriveTeach‑VLA, the first MLLM‑native autonomous driving model that tackles the mismatch between language‑centric VLA training data and the action‑oriented planning required for self‑driving. Existing VLA models rely heavily on visual‑question‑answering and chain‑of‑thought data, which focus on language reasoning rather than the visual elements crucial for safe trajectory generation.

To address this, the authors propose two key innovations. First, Driving‑aware Vision Distillation (DVD) injects traffic‑relevant visual supervision by using GroundingDINO to annotate bounding boxes for vehicles, pedestrians, obstacles, and traffic lights. These annotations are overlaid on the images and fed to a teacher ViT, while a student ViT processes the raw image; the teacher’s attention, guided by the boxes, is distilled to the student via block‑level perception distillation similar to Swin Transformer.

Second, the 2D Trajectory‑Guided Prompt (2D‑TGP) maps expert BEV trajectories into image‑pixel coordinates using camera intrinsics and extrinsics, then formats the pixel sequence as textual prompts. This provides the VLA with concrete visual references of where the vehicle should go, linking planning space with the model’s strong image‑reading capabilities.

DriveTeach‑VLA employs two Qwen2.5‑VL‑3B models: a TGP‑Prompter that predicts the 2D‑TGP from the current image, vehicle state, and driving command, and a TGP‑Planner that consumes the image, state, command, and the predicted 2D‑TGP to output the final BEV trajectory. During training, the Planner receives ground‑truth 2D‑TGP (teacher forcing), while at inference it uses the Prompter’s prediction, and the authors verify that minor errors in the image trajectory do not catastrophically affect the final BEV prediction.

Extensive ablations show that the baseline Qwen2.5‑VL‑3B achieves 84.8 PDMS; adding VQA and chain‑of‑thought raises it to 86.4, DVD further to 87.3, and 2D‑TGP to 88.2. With GRPO behavior alignment, the full system reaches 90.4 PDMS and 85.4 EPDMS on NAVSIM, and an L2 error of 0.3 on nuScenes. The improvements stem from the supervision strategy rather than model size, as the Planner remains a 3B model.

When generating multiple candidate trajectories (12 per scene) and selecting the best with the DrivorR selector, performance climbs to 92.7 PDMS and 89.0 EPDMS, suggesting further gains are possible via techniques like model distillation. The authors conclude that DriveTeach‑VLA provides a practical bridge between BEV trajectory planning and the visual reasoning strengths of large multimodal models, offering a scalable blueprint for future VLA research.