MOJITO让自动驾驶模型直接联合感知与决策，提升规划精度与多样性。
MOJITO: Modal Joint Learning for Unified End-to-End Autonomous Driving
- 采用块级模态联合注意力，同步更新动作、图像与激光雷达特征
- 在NAVSIM v1和v2上分别达到88.9和88.4 EPDMS新高
- 支持指令跟随与多样化轨迹生成，适合复杂场景驾驶
端到端自动驾驶系统通常采用级联的两阶段流程：感知阶段将多模态传感器输入压缩为紧凑上下文，规划器基于该上下文预测轨迹。我们指出，这种单向感知到规划的接口迫使传感器输入进入紧凑表示，丢失对规划至关重要的细粒度信息；同时，受限于压缩上下文，难以利用现代视觉基础模型提供的丰富表征。为此，我们提出MOJITO，一种基于模态联合学习的统一传感器到动作框架。MOJITO移除级联接口，采用块级模态联合注意力机制，同时更新动作、图像与激光雷达特征，使规划器在生成动作时可直接访问多模态特征。MOJITO在NAVSIM v1数据集上取得88.9 PDMS，在更具挑战性的NAVSIM v2数据集上达到88.4 EPDMS，创下新纪录。大量实验进一步验证其出色的可扩展性、指令遵循能力与多样轨迹生成能力。代码与模型已开源。
原文摘要 · Abstract (English)
End-to-end autonomous driving systems commonly follow a cascaded two-stage pipeline where a perception stage compresses multi-modal sensor inputs into a compact context and a downstream planner predicts trajectories conditioned on this context. We argue that this one-way perception-to-planning interface forces sensor inputs into a compact representation, losing the fine-grained details critical for planning. Moreover, by constraining the planner to this compressed context, it is difficult to leverage the rich representations offered by modern vision foundation models. To address these issues, we propose MOJITO, a unified sensor-to-action framework for end-to-end autonomous driving built on modal joint learning. MOJITO removes the cascaded interface and instead performs block-wise Modal Joint Attention that simultaneously updates action, image, and LiDAR features, allowing the planner to directly access multi-modal features during action generation. MOJITO achieves 88.9 PDMS on the NAVSIM v1 dataset and 88.4 EPDMS on the more challenging NAVSIM v2 dataset, setting a new state-of-the-art. Extensive experiments further demonstrate strong scalability, instruction following, and diverse trajectory generation. Code and models are available at https://github.com/mumucc01/MOJITO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。