提出多模态协同机制,提升自动驾驶决策的可靠性与可解释性。
A Collaborative Multi-Modality Interaction for VLA-based End-to-End Autonomous Driving

- 通过双向交互优化主辅模态信息融合
- 在长尾场景下实现更安全的多轨迹规划与感知
- 适合关注自动驾驶系统可解释性的研究者
视觉-语言-动作(VLA)模型通过统一多模态框架整合感知、推理与决策,成为端到端自动驾驶的有力范式。然而,现有方法多将自动驾驶视为视觉问答任务,导致决策推理不可靠且缺乏可解释性;同时,异构传感器间的多模态交互不足,限制了复杂场景下的鲁棒感知与可靠推理。为此,本文提出一种基于多模态协同与多轨迹规划优化的鲁棒端到端自动驾驶系统。核心包括:(1) 基于亲和力引导的最优传输实现主-辅模态双向交互;(2) 分布一致性模态迁移实现异构模态分布对齐与跨模态交互;(3) 多模态多轨迹规划结合感知导向的轨迹精修,提升长尾场景下的驾驶决策能力。在开环与闭环数据集上的实验表明,系统在长时程驾驶推理与道路场景感知方面优于现有方法,验证了所提多模态交互与多轨迹规划优化在可扩展VLA系统中的有效性。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models have emerged as a powerful paradigm for end-to-end autonomous driving by jointly integrating perception, reasoning, and decision making within a unified multimodal framework. However, most existing VLA models formulate end-to-end autonomous driving as a visual question answering task, leading to unreliable and less interpretable decision reasoning. In addition, they fail to establish effective multi-modal interaction across heterogeneous sensors, thereby limiting robust scene perception and reliable driving reasoning in long-tail driving scenarios. To this end, we propose a robust VLA-based end-to-end autonomous driving system that combines multi-modality interaction with multi-trajectory planning and optimization, enabling more reliable, interpretable, and safer driving decisions. Our method comprises three core components: (1) Affinity-Guided Optimal Transport for main-auxiliary modality two-way interaction; (2) Distribution-Consistent Modality Transfer for heterogeneous modality distribution transfer and cross-modal interaction; (3) Multi-modal Multi-Trajectory Planning along with Perception-Oriented Trajectory Refinement for better driving decisions to long-tail driving scenarios. Experimental results in open-loop and closed-loop datasets demonstrate improvements in safety long-horizon driving reasoning and road scene perception over existing driving systems, highlighting the ability of our mutli-modality interaction and multi-trajectory planning and optimization for scalable VLA-based systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。