EmoVerse让大模型同时理解情感与情绪,提升微妙表情识别能力。
EmoVerse: Exploring Multimodal Large Language Models for Sentiment and Emotion Understanding
- 采用多任务训练策略,统一处理情感与情绪任务
- 在5项任务上达领先效果,显著优于现有方法
- 适合人机交互、抑郁检测等需要深度情绪理解的场景
情感与情绪理解对人机交互、抑郁症检测等应用至关重要。尽管多模态大语言模型(MLLMs)具备强大泛化能力,但在情感计算领域仍面临挑战,尤其在识别细微面部表情及处理复杂情绪任务(如情绪原因推理、长上下文情绪理解)方面表现不足。此外,缺乏能统一处理情感与情绪任务的通用型MLLM。为此,本文探索了情感计算中MLLM的多任务训练策略,提出EmoVerse——一个面向广泛情感与情绪任务的多模态大语言模型,并具备深入分析情绪成因的能力。同时,我们构建了支持多模态情感分析、多模态情绪识别、面部表情识别、情绪原因推理及情绪因果对提取的Affective Multitask(AMT)数据集。大量实验表明,EmoVerse在多项任务上达到最先进水平。代码已开源:https://github.com/liaolea/EmoVerse。
原文摘要 · Abstract (English)
Sentiment and emotion understanding are essential to applications such as human-computer interaction and depression detection. While Multimodal Large Language Models (MLLMs) demonstrate robust general capabilities, they face considerable challenges in the field of affective computing, particularly in detecting subtle facial expressions and handling complex emotion-related tasks, such as emotion reason inference and understanding emotions in long-context scenarios. Furthermore, there is a lack of a unified MLLM that can effectively handle both sentiment and emotion-related tasks. To address these challenges, we explore multi-task training strategies for MLLMs in affective computing and introduce Emotion Universe (EmoVerse), an MLLM designed to handle a broad spectrum of sentiment and emotion-related tasks. In addition, EmoVerse is capable of deeply analyzing the underlying causes of emotional states. We also introduce the Affective Multitask (AMT) Dataset, which supports multimodal sentiment analysis, multimodal emotion recognition, facial expression recognition, emotion reason inference, and emotion cause-pair extraction tasks. Extensive experiments demonstrate that EmoVerse outperforms existing methods, achieving state-of-the-art results in sentiment and emotion-related tasks. The code is available at https://github.com/liaolea/EmoVerse.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。