统一视觉语言动作模型,实现端到端自动驾驶的高效推理与决策。
AutoMoT: A Unified Vision-Language-Action Model with Asynchronous Mixture-of-Transformers for End-to-End Autonomous Driving
- 采用异步混合Transformer架构,分频执行推理与决策任务。
- 仅用语义提示即达多任务场景理解性能,但决策需微调。
- 兼顾通用性与效率,适合追求实时性的自动驾驶系统研发。
将视觉语言模型(VLM)融入端到端自动驾驶(AD)系统在提升场景理解方面展现出潜力。然而,现有方法存在推理与动作空间分布不一致、未充分利用预训练VLM的通用推理能力,以及动作策略生成时延迟高等问题。为此,本文提出AutoMoT,一个统一视觉-语言-动作(VLA)的端到端自动驾驶框架。该方法基于具有联合注意力共享的混合变压器(MoT)架构,既保留了预训练VLM的通用推理能力,又通过异步执行实现快慢任务分频处理,提升推理效率。在多个基准测试中,无论开环还是闭环设置下,AutoMoT均达到与先进方法相当的性能。进一步研究发现,仅通过语义提示即可使预训练VLM在多任务场景理解上表现良好,但在决策与轨迹规划等动作级任务上仍需针对自动驾驶进行微调。更多演示视频与定性结果见https://automot-website.github.io/。
原文摘要 · Abstract (English)
Integrating vision-language models (VLMs) into end-to-end (E2E) autonomous driving (AD) systems has shown promise in improving scene understanding. However, existing integration strategies suffer from several limitations: they either struggle to resolve distribution misalignment between reasoning and action spaces, underexploit the general reasoning capabilities of pretrained VLMs, or incur substantial inference latency during action policy generation, which degrades driving performance. To address these challenges, we propose AutoMoT in this work, an end-to-end AD framework that unifies reasoning and action generation within a single vision-language-action (VLA) model. Our approach leverages a mixture-of-transformer (MoT) architecture with joint attention sharing, which preserves the general reasoning capabilities of pre-trained VLMs while enabling efficient fast-slow inference through asynchronous execution at different task frequencies. Extensive experiments on multiple benchmarks, under both open- and closed-loop settings, demonstrate that AutoMoT achieves competitive performance compared to state-of-the-art methods. We further investigate the functional boundary of pre-trained VLMs in AD, examining when AD-tailored fine-tuning is necessary. Our results show that pre-trained VLMs can achieve competitive multi-task scene understanding performance through semantic prompting alone, while fine-tuning remains essential for action-level tasks such as decision-making and trajectory planning. We refer to https://automot-website.github.io/ for the demonstration videos and qualitative results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。