将目标检测模块融入视觉语言模型，提升自动驾驶环境理解能力
Integrating Object Detection Modality into Visual Language Model for Enhanced Autonomous Driving Agent
- 用YOLO-S检测网络增强CLIP感知，融合多模态视觉信息
- 在DriveLM挑战中问答得分显著优于基线模型
- 适合关注自动驾驶视觉理解与安全性的研究者
本文提出一种新框架，通过将专用于目标检测的视觉感知模块集成到视觉语言模型（VLM）中，以增强自动驾驶系统的视觉理解能力。我们扩展了Llama-Adapter架构，引入基于YOLOS的检测网络与原有的CLIP感知网络并行工作，弥补了传统方法在物体检测与定位上的不足。通过引入相机ID分隔器，有效提升了多视角处理能力，有助于实现更全面的环境感知。在DriveLM视觉问答挑战上的实验表明，该方法在ChatGPT评分、BLEU分数和CIDEr指标上均显著优于基线模型，说明生成答案更接近真实答案。研究还讨论了检测模态带来的潜在安全提升可能性。
原文摘要 · Abstract (English)
In this paper, we propose a novel framework for enhancing visual comprehension in autonomous driving systems by integrating visual language models (VLMs) with additional visual perception module specialised in object detection. We extend the Llama-Adapter architecture by incorporating a YOLOS-based detection network alongside the CLIP perception network, addressing limitations in object detection and localisation. Our approach introduces camera ID-separators to improve multi-view processing, crucial for comprehensive environmental awareness. Experiments on the DriveLM visual question answering challenge demonstrate significant improvements over baseline models, with enhanced performance in ChatGPT scores, BLEU scores, and CIDEr metrics, indicating closeness of model answer to ground truth. Our method represents a promising step towards more capable and interpretable autonomous driving systems. Possible safety enhancement enabled by detection modality is also discussed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。