用视觉语言模型实现单目端到端自动驾驶，性能领先
2nd Place Solution for CVPR2024 E2E Challenge: End-to-End Autonomous Driving Using Vision Language Model
- 结合端到端架构与多模态视觉语言模型
- 仅用单摄像头即达榜单最佳表现
- 适合关注视觉驱动自动驾驶的研究者
端到端自动驾驶近年备受关注，多数工作采用模块化深度神经网络构建系统。然而，强大大型语言模型（尤其是多模态视觉语言模型，VLM）是否能提升端到端驾驶任务仍不明确。本文证明，将端到端架构与知识丰富的VLM结合，可在驾驶任务上取得优异表现。值得注意的是，本方法仅使用单个摄像头，在排行榜中成为最佳纯视觉方案，验证了基于视觉的驾驶方法的有效性及端到端任务的潜力。
原文摘要 · Abstract (English)
End-to-end autonomous driving has drawn tremendous attention recently. Many works focus on using modular deep neural networks to construct the end-to-end archi-tecture. However, whether using powerful large language models (LLM), especially multi-modality Vision Language Models (VLM) could benefit the end-to-end driving tasks remain a question. In our work, we demonstrate that combining end-to-end architectural design and knowledgeable VLMs yield impressive performance on the driving tasks. It is worth noting that our method only uses a single camera and is the best camera-only solution across the leaderboard, demonstrating the effectiveness of vision-based driving approach and the potential for end-to-end driving tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。