用多模态大模型实现情绪驱动的共情回复，保持身份一致且无需额外训练。
E3RG: Building Explicit Emotion-driven Empathetic Response Generation System with Multimodal Large Language Model
- 分三步：理解多模态情绪、检索共情记忆、生成多模态回复。
- 在零样本和少样本下均领先，获ACM MM 25虚拟形象共情挑战赛第一名。
- 支持语音与视频生成，自然流畅且身份一致性高，适合智能客服与虚拟助手。
多模态共情回复（MERG）对构建情感智能人机交互至关重要。尽管大语言模型（LLMs）提升了文本共情回复能力，但在处理多模态情感内容和保持身份一致性方面仍面临挑战。为此，我们提出E3RG系统，基于多模态大模型将MERG任务分解为三部分：多模态共情理解、共情记忆检索与多模态回复生成。通过集成先进的语音与视频生成模型，E3RG在无需额外训练的情况下，实现自然、情感丰富且身份一致的回复。实验验证了该系统在零样本与少样本设置下的优越性，在ACM MM 25的基于虚拟形象的多模态共情挑战赛中获得第一名。代码已开源于https://github.com/RH-Lin/E3RG。
原文摘要 · Abstract (English)
Multimodal Empathetic Response Generation (MERG) is crucial for building emotionally intelligent human-computer interactions. Although large language models (LLMs) have improved text-based ERG, challenges remain in handling multimodal emotional content and maintaining identity consistency. Thus, we propose E3RG, an Explicit Emotion-driven Empathetic Response Generation System based on multimodal LLMs which decomposes MERG task into three parts: multimodal empathy understanding, empathy memory retrieval, and multimodal response generation. By integrating advanced expressive speech and video generative models, E3RG delivers natural, emotionally rich, and identity-consistent responses without extra training. Experiments validate the superiority of our system on both zero-shot and few-shot settings, securing Top-1 position in the Avatar-based Multimodal Empathy Challenge on ACM MM 25. Our code is available at https://github.com/RH-Lin/E3RG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。