让自动驾驶模型通过多轮交互迭代优化行驶路径，提升复杂场景应对能力。
MTDrive: Multi-turn Interactive Reinforcement Learning for Autonomous Driving
- 采用多轮交互式强化学习，让模型基于环境反馈逐步优化行驶轨迹。
- 在NAVSIM上表现优于现有方法，训练效率提升2.5倍。
- 适合研究自动驾驶规划与多轮推理的学者和工程师。
轨迹规划是自动驾驶的核心任务，需在多样场景中预测安全舒适的路径。将多模态大语言模型（MLLM）与强化学习（RL）结合，在处理长尾场景方面展现出潜力。然而，现有方法仅支持单轮推理，难以应对需要迭代优化的复杂任务。为此，我们提出MTDrive，一种多轮交互框架，使MLLM能基于环境反馈迭代优化轨迹。MTDrive引入多轮组相对策略优化（mtGRPO），通过跨轮次计算相对优势缓解奖励稀疏问题。我们还构建了基于闭环仿真的交互式轨迹理解数据集，支持多轮训练。在NAVSIM基准上的实验表明，该方法性能优于现有方案。此外，通过系统级优化，解决了高分辨率图像与多轮序列带来的数据传输开销，训练吞吐量提升2.5倍。相关数据、模型与代码即将开源。
原文摘要 · Abstract (English)
Trajectory planning is a core task in autonomous driving, requiring the prediction of safe and comfortable paths across diverse scenarios. Integrating Multi-modal Large Language Models (MLLMs) with Reinforcement Learning (RL) has shown promise in addressing "long-tail" scenarios. However, existing methods are constrained to single-turn reasoning, limiting their ability to handle complex tasks requiring iterative refinement. To overcome this limitation, we present MTDrive, a multi-turn framework that enables MLLMs to iteratively refine trajectories based on environmental feedback. MTDrive introduces Multi-Turn Group Relative Policy Optimization (mtGRPO), which mitigates reward sparsity by computing relative advantages across turns. We further construct an interactive trajectory understanding dataset from closed-loop simulation to support multi-turn training. Experiments on the NAVSIM benchmark demonstrate superior performance compared to existing methods, validating the effectiveness of our multi-turn reasoning paradigm. Additionally, we implement system-level optimizations to reduce data transfer overhead caused by high-resolution images and multi-turn sequences, achieving 2.5x training throughput. Our data, models, and code will be made available soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。