构建视觉因果推理基准，评估大模型在图像中的因果理解能力。
CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Models
- 设计三类视觉因果任务：结构推断、干预预测、反事实推理。
- 在三个数据集上测试主流开源模型，发现其因果推理能力有限。
- 适合关注多模态模型认知能力与改进方向的研究者使用。
大型语言模型（LLMs）在多种语言任务中表现出色，尤其具备上下文学习的涌现能力。将LLMs扩展至包含视觉输入的大规模视觉-语言模型（LVLMs），在识别和视觉问答（VQA）等任务中也展现出优异性能。尽管人们对LLMs在因果推理任务（如因果发现和反事实推理）中的应用日益关注，但针对LVLM在视觉因果推理任务上的表现研究仍相对不足。本文正式提出一个面向多模态上下文学习的综合性视觉因果推理基准——CausalVLBench，涵盖三类代表性任务：因果结构推断、干预目标预测和反事实预测。我们在三个因果表征学习数据集上评估了当前最先进的开源LVLM在这些任务中的表现，揭示了其基础优势与局限性。我们希望该基准能阐明现有视觉-语言模型的缺陷，并推动提升LVLM视觉因果推理能力的新方向与新范式。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown remarkable ability in various language tasks, especially with their emergent in-context learning capability. Extending LLMs to incorporate visual inputs, large vision-language models (LVLMs) have shown impressive performance in tasks such as recognition and visual question answering (VQA). Despite increasing interest in the utility of LLMs in causal reasoning tasks such as causal discovery and counterfactual reasoning, there has been relatively little work showcasing the abilities of LVLMs on visual causal reasoning tasks. We take this opportunity to formally introduce a comprehensive causal reasoning benchmark for multi-modal in-context learning from LVLMs. Our CausalVLBench encompasses three representative tasks: causal structure inference, intervention target prediction, and counterfactual prediction. We evaluate the ability of state-of-the-art open-source LVLMs on our causal reasoning tasks across three causal representation learning datasets and demonstrate their fundamental strengths and weaknesses. We hope that our benchmark elucidates the drawbacks of existing vision-language models and motivates new directions and paradigms in improving the visual causal reasoning abilities of LVLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。