arXiv:2605.04647cs.RO2026-05被引 2

用可自修正的离散轨迹令牌,让自动驾驶规划更智能高效。

ReflectDrive-2: Reinforcement-Learning-Aligned Self-Editing for Discrete Diffusion Driving

论文配图:ReflectDrive-2: Reinforcement-Learning-Aligned Self-Editing for Discrete Diffusion Driving
图 1 · 摘自论文原文
  • 将驾驶规划建模为离散令牌序列,支持推理时直接修改轨迹。
  • 结合强化学习训练,使修正后轨迹性能提升1.9点PDMS。
  • 适合追求高精度、低延迟自动驾驶系统的研究与开发。

我们提出ReflectDrive-2,一种基于掩码离散扩散的自动驾驶规划器,将路径表示为离散轨迹令牌,并通过并行掩码解码生成。该离散空间支持原位轨迹修订:AutoEdit使用同一模型重写选定令牌,无需额外精修网络。训练采用两阶段流程:首先构建沿纵向进度和横向朝向的结构感知扰动轨迹,并监督模型恢复原始专家轨迹;随后在完整决策-起草-反思流水线中进行强化学习微调,对最终修正轨迹赋予终端驾驶奖励,并通过全回滚转移传播策略梯度。全回滚强化学习对耦合起草与编辑至关重要:仅监督训练下,推理时AutoEdit最多提升PDMS 0.3,而强化学习将其提升至1.9。我们还协同设计了高效的反思解码堆栈,结合共享前缀KV复用、交替步解码和融合式设备内解掩码。在NAVSIM上,ReflectDrive-2以纯摄像头输入达成91.0 PDMS,最佳6选1设置下达94.8 PDMS,平均延迟仅31.8毫秒(NVIDIA Thor)。

原文摘要 · Abstract (English)

We introduce ReflectDrive-2, a masked discrete diffusion planner with separate action expert for autonomous driving that represents plans as discrete trajectory tokens and generates them through parallel masked decoding. This discrete token space enables in-place trajectory revision: AutoEdit rewrites selected tokens using the same model, without requiring an auxiliary refinement network. To train this capability, we use a two-stage procedure. First, we construct structure-aware perturbations of expert trajectories along longitudinal progress and lateral heading directions and supervise the model to recover the original expert trajectory. We then fine-tune the full decision--draft--reflect rollout with reinforcement learning (RL), assigning terminal driving reward to the final post-edit trajectory and propagating policy-gradient credit through full-rollout transitions. Full-rollout RL proves crucial for coupling drafting and editing: under supervised training alone, inference-time AutoEdit improves PDMS by at most $0.3$, whereas RL increases its gain to $1.9$. We also co-design an efficient reflective decoding stack for the decision--draft--reflect pipeline, combining shared-prefix KV reuse, Alternating Step Decode, and fused on-device unmasking. On NAVSIM, ReflectDrive-2 achieves $91.0$ PDMS with camera-only input and $94.8$ PDMS in a best-of-6 oracle setting, while running at $31.8$ ms average latency on NVIDIA Thor.

自动驾驶扩散模型强化学习轨迹规划

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。