arXiv:2509.02100cs.HCcs.CL2025-09被引 1

构建首个关注情感不一致的多模态数据集,让AI真正理解情绪背后的真实状态。

E-THER: A Multimodal Dataset for Empathic AI -- Towards Emotional Mismatch Awareness

  • 基于心理咨询场景,标注言语与表情不一致的多维度情感错配
  • 使用该数据训练的模型在共情和治疗对话质量上显著提升
  • 适合研究真实共情、人机交互与心理智能的团队使用

当前共情型AI系统普遍无法识别言语表达与真实情绪状态之间的偏差,根源在于现有数据集仅关注表面情绪识别,缺乏对言语与视觉信息不一致模式的建模。本文提出E-THER,首个基于个人中心疗法(PCT)的多模态数据集,包含多维标注以支持言语-视觉情感错配检测,推动具备真实共情能力而非表演性回应的AI发展。标注依据人本主义方法,聚焦来访者与咨询师互动中的情绪不一致现象,形成可评估共情任务的框架。附加的参与度评分提供行为标注,适用于多类研究。实验表明,采用E-THER训练的先进视觉-语言模型(如IDEFICS和VideoLLAVA)在基于共情与治疗原则的评估中表现显著提升。实证结果表明,经错配训练的模型在维持治疗连贯性、减少人为夸张语言模式以及符合PCT理论框架方面优于通用模型。

原文摘要 · Abstract (English)

A prevalent shortfall among current empathic AI systems is their inability to recognize when verbal expressions may not fully reflect underlying emotional states. This is because the existing datasets, used for the training of these systems, focus on surface-level emotion recognition without addressing the complex verbal-visual incongruence (mismatch) patterns useful for empathic understanding. In this paper, we present E-THER, the first Person-Centered Therapy-grounded multimodal dataset with multidimensional annotations for verbal-visual incongruence detection, enabling training of AI systems that develop genuine rather than performative empathic capabilities. The annotations included in the dataset are drawn from humanistic approach, i.e., identifying verbal-visual emotional misalignment in client-counsellor interactions - forming a framework for training and evaluating AI on empathy tasks. Additional engagement scores provide behavioral annotations for research applications. Notable gains in empathic and therapeutic conversational qualities are observed in state-of-the-art vision-language models (VLMs), such as IDEFICS and VideoLLAVA, using evaluation metrics grounded in empathic and therapeutic principles. Empirical findings indicate that our incongruence-trained models outperform general-purpose models in critical traits, such as sustaining therapeutic engagement, minimizing artificial or exaggerated linguistic patterns, and maintaining fidelity to PCT theoretical framework.

共情AI多模态情感错配心理智能

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。