首个评估多模态模型社会推理能力的基准，聚焦真实交互中的细粒度理解。
Social Genome: Grounded Social Reasoning Abilities of Multimodal Models
- 构建272段真实互动视频与1486条人类标注的推理链，涵盖视觉、语言、语音及外部知识。
- 包含5777个推理步骤，首次系统评估模型利用外部知识进行社会推理的能力。
- 适合研究多模态理解、社会认知与可解释性的人工智能学者使用。
社会推理能力对AI系统理解并回应多模态人际交流与社会互动至关重要。我们提出SOCIAL GENOME，首个针对多模态模型细粒度、有根基的社会推理能力的基准。该数据集包含272段互动视频和1,486条人类标注的推理轨迹，涉及对这些互动的推断。这些轨迹包含5,777个推理步骤，引用视觉线索、语言线索、语音线索及外部知识（视频外的上下文知识）。SOCIAL GENOME也是首个研究社会推理中外部知识作用的建模范式。它通过综合指标评估模型生成的社会推理轨迹在语义与结构上的质量。我们通过与前沿模型的实验展示了SOCIAL GENOME的实用性，揭示了当前模型在有根基社会推理上的性能差距，并为未来研究指明方向。
原文摘要 · Abstract (English)
Social reasoning abilities are crucial for AI systems to effectively interpret and respond to multimodal human communication and interaction within social contexts. We introduce SOCIAL GENOME, the first benchmark for fine-grained, grounded social reasoning abilities of multimodal models. SOCIAL GENOME contains 272 videos of interactions and 1,486 human-annotated reasoning traces related to inferences about these interactions. These traces contain 5,777 reasoning steps that reference evidence from visual cues, verbal cues, vocal cues, and external knowledge (contextual knowledge external to videos). SOCIAL GENOME is also the first modeling challenge to study external knowledge in social reasoning. SOCIAL GENOME computes metrics to holistically evaluate semantic and structural qualities of model-generated social reasoning traces. We demonstrate the utility of SOCIAL GENOME through experiments with state-of-the-art models, identifying performance gaps and opportunities for future research to improve the grounded social reasoning abilities of multimodal models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。