统一视觉与动作生成,让自动驾驶模型先决策再预测未来画面。
ForgeDrive: Bidirectional Cross-Conditioning for Unified Visual-Action Generation in Autonomous Driving

- 采用动作-视觉交叉条件机制,实现双向信息交互。
- 在NAVSIM上超越现有强基线规划器,无需后处理优化。
- 创新提出‘先动后想’推理范式,提升规划与视觉生成质量。
基于世界模型的自动驾驶赋予模型理解场景演化的潜力,但当前主流的‘先想象后行动’范式导致视觉生成阶段的误差会传递至动作规划阶段。本文提出ForgeDrive,一种统一的自回归扩散框架,通过‘先行动后想象’的新范式弥补这一缺陷。该框架将未来建模为逐时步的帧-动作对序列,使每个动作与其对应视觉观测紧密耦合。训练中,解耦两模态的扩散步数,并引入类似UniDiffuser的噪声调度器,实现从一模态推断另一模态的能力,深化图像与动作间的关系理解。推理时,提出新型‘先动后想’策略:每一步的动作生成是训练中内化的能力,无需干净的未来帧作为前提;相反,生成的动作可提升未来帧预测精度,进而反哺后续动作质量。此外,每一步还融合未来自身状态预测,进一步增强规划能力。大量实验表明,ForgeDrive将驾驶仿真、规划与视觉里程计统一于单一模型,在NAVSIM上表现优于现有强基线规划器,且无需任何后训练策略。
原文摘要 · Abstract (English)
World-model-based autonomous driving endows the model with the ability to understand scene evolution. Yet this promise is undermined by the prevailing imagine-then-act paradigm, which allows errors from the more challenging visual generation stage to cascade into action planning. We introduce ForgeDrive, a unified autoregressive diffusion framework with visual-action cross-conditioning that closes this gap through act-then-imagine paradigm. ForgeDrive factorizes the future as a sequence of per-timestep frame-action pairs, intertwining each action with its corresponding visual observation. During training, we decouple the diffusion timesteps of the two modalities and introduce a UniDiffuser-style noise scheduler to get the ability to infer either modality from its counterpart and deepen understanding of relationships between images and actions. At inference, we propose a novel act-then-imagine inference paradigm, and find that at each step, action generation is a capability internalized during training, requiring no clean future frame as a prerequisite at inference time; instead, the generated action can improve the accuracy of future frame generation, which in turn enhances the quality of the next action. Additionally, we augment each step with future ego-status prediction, further sharpening planning ability. Extensive experiments on NAVSIM demonstrate that ForgeDrive not only unifies driving simulation, planning, and visual odometry into a single model, but also outperforms existing strong planners without any post-training strategy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。