用视觉片段提升自动驾驶场景下的模型推理能力
Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarios
- 通过检索相关视觉区域，让模型分步推理
- 在驾驶考题上准确率提升3.1%，推理正确率升4.6%
- 适合需要精准视觉理解的自动驾驶研究者
尽管思维链（CoT）提示能提升大语言模型的推理能力，但视觉语言模型（VLMs）在复杂现实场景中仍受限于对文本线索的过度依赖和记忆知识。为探究VLM在真实驾驶场景中的视觉推理能力，我们构建了DrivingVQA数据集，包含3,931道来自驾驶理论考试的多选题，附有专家撰写解析及与推理相关的实体。基于此，我们提出检索式交错视觉思维链方法（RIV-CoT），使VLM能利用对应实体的视觉切片进行推理。实验表明，RIV-CoT相比原始CoT提示，答案准确率提升3.1%，推理准确率提升4.6%。此外，该方法可扩展至更大规模的A-OKVQA推理数据集，通过自动伪标签实现性能超越传统CoT提示。
原文摘要 · Abstract (English)
While chain-of-thought (CoT) prompting improves reasoning in large language models, its effectiveness in vision-language models (VLMs) remains limited due to over-reliance on textual cues and memorized knowledge. To investigate the visual reasoning capabilities of VLMs in complex real-world scenarios, we introduce DrivingVQA, a visual question answering dataset derived from driving theory exams, which contains 3,931 multiple-choice problems with expert-written explanations and grounded entities relevant to the reasoning process. Leveraging this dataset, we propose RIV-CoT, a Retrieval-Based Interleaved Visual Chain-of-Thought method that enables VLMs to reason using visual crops corresponding to these relevant entities. Our experiments demonstrate that RIV-CoT improves answer accuracy by 3.1% and reasoning accuracy by 4.6% over vanilla CoT prompting. Furthermore, we demonstrate that our method effectively scales to the larger A-OKVQA reasoning dataset by leveraging automatically generated pseudo-labels, outperforming CoT prompting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。