CORAL: Evaluating Chain-of-Thought Reasoning in Vision–Language Models for End-to-End Autonomous Driving

Abstract

Vision–language models (VLMs) provide a unified framework for end-to-end autonomous driving by combining visual perception with semantic reasoning. A key question is whether intermediate reasoning representations, such as Chain-of-Thought (CoT), improve trajectory prediction in continuous control settings. This thesis presents CORAL, a controlled study of CoT supervision in VLM-based driving models on the Waymo Open Dataset End-to-End Driving (WOD-E2E), which emphasizes rare, safety-critical long-tail scenarios. Qwen2.5-VL-3B, an off-the-shelf VLM, is fine-tuned with and without CoT under identical supervised fine-tuning (SFT) conditions, followed by reinforcement learning via group relative policy optimization (GRPO). Rater Feedback Score (RFS) is used as both the reward signal and the evaluation metric. While CoT improves training dynamics, including faster convergence and lower loss, it does not improve trajectory quality. After GRPO, the CoT and No-CoT variants achieve nearly identical performance (RFS 7.96 vs. 7.97) on the WOD-E2E test set. Qualitative results show strong performance in simple scenarios, with errors concentrated in high-curvature and interaction-heavy cases. Overall, intermediate language reasoning does not directly benefit trajectory prediction, while RL-based alignment plays a more decisive role in final performance. This thesis discusses implications for supervision design, reward modeling, and evaluation in long-tail driving scenarios.