轻量级多模态模型提升自动驾驶语言理解能力
VLDrive: Vision-Augmented Lightweight MLLMs for Efficient Language-grounded Autonomous Driving
- 用动态视觉剪枝和记忆增强聚合压缩视觉特征
- 参数减少81%（7B→1.3B），长距离驾驶得分提升7.6%
- 适合资源受限场景的智能驾驶系统开发
语言引导的自动驾驶近年得益于大语言模型（LLM）的认知与推理能力。然而现有方法仍面临两大挑战：（1）视觉表征不足导致频繁碰撞与遮挡；（2）LLM参数庞大，部署困难。为此，我们提出VLDrive，一种轻量级多模态大模型架构，通过循环一致的动态视觉剪枝与记忆增强特征聚合，实现紧凑视觉标记。同时引入距离解耦指令注意力机制，强化远距离视觉标记的跨模态学习。在CARLA模拟器中大量实验表明，VLDrive在保持状态领先性能的同时，参数量降低81%（从70亿降至13亿），在闭环评估中于短、中、长距离分别获得15.4%、16.8%和7.6%的驾驶得分提升。代码已开源。
原文摘要 · Abstract (English)
Recent advancements in language-grounded autonomous driving have been significantly promoted by the sophisticated cognition and reasoning capabilities of large language models (LLMs). However, current LLM-based approaches encounter critical challenges: (1) Failure analysis reveals that frequent collisions and obstructions, stemming from limitations in visual representations, remain primary obstacles to robust driving performance. (2) The substantial parameters of LLMs pose considerable deployment hurdles. To address these limitations, we introduce VLDrive, a novel approach featuring a lightweight MLLM architecture with enhanced vision components. VLDrive achieves compact visual tokens through innovative strategies, including cycle-consistent dynamic visual pruning and memory-enhanced feature aggregation. Furthermore, we propose a distance-decoupled instruction attention mechanism to improve joint visual-linguistic feature learning, particularly for long-range visual tokens. Extensive experiments conducted in the CARLA simulator demonstrate VLDrive`s effectiveness. Notably, VLDrive achieves state-of-the-art driving performance while reducing parameters by 81% (from 7B to 1.3B), yielding substantial driving score improvements of 15.4%, 16.8%, and 7.6% at tiny, short, and long distances, respectively, in closed-loop evaluations. Code is available at https://github.com/ReaFly/VLDrive.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。