提出并行框架让模仿与强化学习协同优化,突破自动驾驶性能瓶颈。
Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Driving
- 设计并行双分支结构,让模仿学习与强化学习独立训练、协同优化
- 在NAVSIMv1/v2上分别达91.2和87.9的PDMS/EPDMS,超越现有方法
- 可修正人类专家错误行为,适合追求高可靠自动驾驶系统的研究者
端到端自动驾驶通常基于模仿学习(IL),但受限于人类示范质量。现有方法通过顺序微调引入强化学习(RL),却易导致策略漂移且性能受限于预训练的IL策略。为此,本文提出PaIR-Drive——一种用于端到端自动驾驶的模仿与强化学习并行协同框架。训练时,将IL与RL拆分为无冲突的并行分支,实现完全协同优化;推理时,RL利用IL策略进一步优化最终轨迹,实现超越原始示范的能力。此外,引入树状轨迹神经采样器与相对策略优化(GRPO),增强探索能力。在NAVSIMv1和v2基准上,基于Transfuser与DiffusionDrive基线,PaIR-Drive分别取得91.2 PDMS和87.9 EPDMS的竞争力表现,持续优于现有微调方法,甚至能纠正人类专家的次优行为。定性结果表明其能有效探索并生成高质量轨迹。
原文摘要 · Abstract (English)
End-to-end autonomous driving is typically built upon imitation learning (IL), yet its performance is constrained by the quality of human demonstrations. To overcome this limitation, recent methods incorporate reinforcement learning (RL) through sequential fine-tuning. However, such a paradigm remains suboptimal: sequential RL fine-tuning can introduce policy drift and often leads to a performance ceiling due to its dependence on the pretrained IL policy. To address these issues, we propose PaIR-Drive, a general Parallel framework for collaborative Imitation and Reinforcement learning in end-to-end autonomous driving. During training, PaIR-Drive separates IL and RL into two parallel branches with conflict-free training objectives, enabling fully collaborative optimization. This design eliminates the need to retrain RL when applying a new IL policy. During inference, RL leverages the IL policy to further optimize the final plan, allowing performance beyond prior knowledge of IL. Furthermore, we introduce a tree-structured trajectory neural sampler to group relative policy optimization (GRPO) in the RL branch, which enhances exploration capability. Extensive analysis on NAVSIMv1 and v2 benchmark demonstrates that PaIR-Drive achieves Competitive performance of 91.2 PDMS and 87.9 EPDMS, building upon Transfuser and DiffusionDrive IL baselines. PaIR-Drive consistently outperforms existing RL fine-tuning methods, and could even correct human experts' suboptimal behaviors. Qualitative results further confirm that PaIR-Drive can effectively explore and generate high-quality trajectories.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。