提升多说话人视频中视觉与语音的对齐,让模型更懂社交互动。
Multi-speaker Attention Alignment for Multimodal Social Interaction
- 动态选择跨模态注意力头,聚焦关键信息
- 无需参数新增,用社交感知偏置增强对齐
- 适配主流多模态大模型,显著提升社交理解
理解视频中的社交互动需综合分析言语与非言语线索:谁在说话、对谁说、眼神或手势如何。尽管多模态大语言模型(MLLMs)是自然选择,但简单添加视觉输入在社交任务上表现不稳。我们对先进MLLMs中跨模态注意力的定量分析发现核心问题:多说话人场景下,视觉与文本标记缺乏说话人一致性对齐,跨模态注意力远弱于物体中心图像。为此,我们提出一种可集成到现有MLLMs的多说话人注意力对齐方法。首先,通过动态跨模态头选择识别最相关注意力头;其次,引入基于已有注意力模式和说话人位置的自适应社交感知偏置,强化说话人视觉表示与其话语间的对齐,无需新增可训练参数或结构改动。我们将该方法集成至三个不同MLLMs(LLaVA-NeXT-Video、Qwen2.5-VL、InternVL3),并在三个基准(TVQA+、MMSI、OnlineMMSI)上评估。在四项社交任务中,结果表明该方法显著提升MLLMs性能并达到当前最优水平。注意力可视化证实模型成功聚焦于说话人相关区域,实现更鲁棒的多方社交推理。代码与模型将公开于https://github.com/ut-vision/SocialInteraction。
原文摘要 · Abstract (English)
Understanding social interaction in video requires reasoning over a dynamic interplay of verbal and non-verbal cues: who is speaking, to whom, and with what gaze or gestures. While Multimodal Large Language Models (MLLMs) are natural candidates, simply adding visual inputs yields surprisingly inconsistent gains on social tasks. Our quantitative analysis of cross-modal attention inside state-of-the-art MLLMs reveals a core failure mode: in multi-speaker scenes, visual and textual tokens lack speaker-consistent alignment, exhibiting substantially weaker cross-modal attention than in object-centric images. To address this, we propose a multimodal multi-speaker attention alignment method that can be integrated into existing MLLMs. First, we introduce dynamic cross-modal head selection to identify attention heads most responsible for grounding. Then, an adaptive social-aware attention bias, computed from existing attention patterns and speaker locations, is injected into the attention mechanism. This bias reinforces alignment between a speaker's visual representation and their utterances without introducing trainable parameters or architectural changes. We integrate our method into three distinct MLLMs (LLaVA-NeXT-Video, Qwen2.5-VL, and InternVL3) and evaluate on three benchmarks (TVQA+, MMSI, OnlineMMSI). Across four social tasks, results demonstrate that our approach improves the ability of MLLMs and achieves state-of-the-art results. Attention visualizations confirm our method successfully focuses the model on speaker-relevant regions, enabling more robust multi-party social reasoning. Our implementation and model will be available at https://github.com/ut-vision/SocialInteraction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。