用视觉大模型提升自动驾驶决策能力，实测接近人类描述水平
Vision-Integrated LLMs for Autonomous Driving Assistance : Human Performance Comparison and Trust Evaluation
- 融合YOLOv4与ViT的视觉适配器提取场景特征
- GPT-4实现类人空间推理，生成响应与人类决策匹配度中等
- 45名驾驶员测试验证系统在描述场景上接近人类表现
传统自动驾驶系统在复杂意外场景中因空间关系理解不足而表现受限。为此，本研究提出一种基于大语言模型（LLM）的自动驾驶辅助系统，集成视觉适配器与LLM推理模块以增强视觉理解与决策能力。视觉适配器结合YOLOv4与视觉变压器（ViT）提取全面视觉特征，GPT-4则实现类人空间推理与响应生成。对45名经验驾驶员的实验评估显示，该系统在情境描述上接近人类表现，在生成适当响应的决策上与人类有中等程度一致。
原文摘要 · Abstract (English)
Traditional autonomous driving systems often struggle with reasoning in complex, unexpected scenarios due to limited comprehension of spatial relationships. In response, this study introduces a Large Language Model (LLM)-based Autonomous Driving (AD) assistance system that integrates a vision adapter and an LLM reasoning module to enhance visual understanding and decision-making. The vision adapter, combining YOLOv4 and Vision Transformer (ViT), extracts comprehensive visual features, while GPT-4 enables human-like spatial reasoning and response generation. Experimental evaluations with 45 experienced drivers revealed that the system closely mirrors human performance in describing situations and moderately aligns with human decisions in generating appropriate responses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。