探究视觉语言模型在社交情境推理中的能力与局限。
VIBE: Can a VLM Read the Room?
- 提出视觉社交语用推理新任务，评估模型理解非语言社交线索能力。
- 发现视觉语言模型在社交语用推理上存在显著能力缺口。
- 构建高质量数据集，为后续研究提供基准评测工具。
理解人类社交行为（如情绪识别及背后的社会动态）是重要且具挑战性的问题。尽管大语言模型在文本领域取得显著进展，但受限于仅处理文本输入，无法充分考虑非语言线索在理解社交情境中的关键作用。视觉语言模型（VLMs）或可弥补这一空白，但其在社交线索推理方面的表现尚未受到足够关注。本文系统探索VLMs在社交推理方面的能力，揭示了一个此前被忽视的缺陷：视觉社交语用推理差距。为此，我们提出一项新任务——视觉社交语用推理，并构建了一个高质量数据集以测试VLMs在此任务上的表现，同时对多个主流VLM进行了基准测试。
原文摘要 · Abstract (English)
Understanding human social behavior such as recognizing emotions and the social dynamics causing them is an important and challenging problem. While LLMs have made remarkable advances, they are limited to the textual domain and cannot account for the major role that non-verbal cues play in understanding social situations. Vision Language Models (VLMs) can potentially account for this gap, however their ability to make correct inferences over such social cues has received little attention. In this paper, we explore the capabilities of VLMs at social reasoning. We identify a previously overlooked limitation in VLMs: the Visual Social-Pragmatic Inference gap. To target this gap, we propose a new task for VLMs: Visual Social-Pragmatic Inference. We construct a high quality dataset to test the abilities of a VLM for this task and benchmark the performance of several VLMs on it.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。