让视觉模型专注说话人表情,提升对话情绪识别准确率。
VISAFF: Speaker-Centered Visual Affective Feature Learning for Emotion Recognition in Conversation

- 冻结大模型,通过聚焦说话人面部区域提取情感视觉特征
- 用语言和语音信息补足视觉模糊部分,提升识别可靠性
- 无需微调,大幅降低计算开销,适合实际部署
对话中的情绪识别(ERC)对人机交互至关重要,旨在识别多轮对话中说话人的情绪状态。早期基于文本的方法在反语等复杂场景下表现不佳,因忽略了关键的非语言信息。尽管近期视觉-语言模型(VLMs)可直接分析视频,但其未针对ERC设计,常关注无关背景或被动听众而非活跃说话人。此外,微调这些大模型成本高昂。孤立的视觉信号在缺乏语言内容和语音韵律背景下往往模糊或失真。为此,我们提出VISAFF框架,包含两个阶段:说话人中心情感定位与可靠性引导的情感补全。VISAFF采用免微调方法,激活冻结VLM的推理能力,高效引导其关注活跃说话人的情感视觉线索,避免高训练开销。第二阶段引入可靠性引导机制,动态利用文本与声学模态补偿视觉不确定性。在两个真实数据集上的实验表明,VISAFF在免微调设置下达到领先性能,显著提升计算效率,无需昂贵的大模型微调。源代码见 https://anonymous.4open.science/r/speaker-2365/。
原文摘要 · Abstract (English)
Emotion Recognition in Conversation (ERC) is essential for effective human-machine interaction, aiming to identify speakers' emotional states in multi-turn dialogues. Early text-based methods struggle with complex scenarios like sarcasm because they inherently neglect vital non-verbal information. While recent Vision-Language Models (VLMs) address this by analyzing video directly, they are not inherently tailored for ERC and often focus on emotionally irrelevant background regions or passive listeners rather than the active speaker. Furthermore, fine-tuning these large models incurs prohibitive computational costs. Additionally, isolated visual signals are frequently ambiguous or technically compromised without the context of linguistic content and vocal prosody. To address these challenges, we propose VISAFF, a speaker-centered VISual AFFective feature learning framework for ERC. VISAFF consists of two stages: Speaker-Centered Affective Grounding and Reliability-Guided Affective Complementation. VISAFF utilizes a tuning-free approach to unlock the reasoning capabilities of frozen VLMs, efficiently steering them to focus on the active speaker's emotional visual cues without heavy training overheads. In the second stage, we introduce a reliability-guided affective complementation mechanism that dynamically leverages textual and acoustic modalities to compensate for visual uncertainty. Experiments on two real-world datasets demonstrate that VISAFF achieves highly competitive performance compared to state-of-the-art methods in a tuning-free setting, significantly enhancing computational efficiency by eliminating the need for expensive fine-tuning of large VLMs. The source code is available at https://anonymous.4open.science/r/speaker-2365/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。