构建首个融合情感与个人特质的多模态对话数据集，助力机器人更懂人心。
MERCI: Multimodal Emotional and peRsonal Conversational Interactions Dataset
- 基于用户画像与情绪识别，用GPT-4生成个性化对话响应
- 自动与人工评估均显示对话自然流畅且具同理心
- 真实人类参与，包含个人经历与真实情绪表达
随着对话代理融入日常生活，其深度交互能力仍显不足。现有数据集缺乏对人机交互中多模态信息的系统记录。为此，我们构建了全新多模态数据集MERCI，涵盖丰富的具身互动数据。参与者首先完成问卷，提供十类个人信息（如兴趣爱好、喜爱音乐）。随后，机器人基于参与者档案及通过面部表情识别与情感分析获得的情绪状态，利用GPT-4生成情境适配回复。经自动与用户评估，对话在自然性、参与感、流畅性、一致性与相关性方面表现优异，且机器人展现出良好共情能力。该数据集源于真实人机互动，参与者提供了真实个人信息并表达了真实情绪。
原文摘要 · Abstract (English)
The integration of conversational agents into our daily lives has become increasingly common, yet many of these agents cannot engage in deep interactions with humans. Despite this, there is a noticeable shortage of datasets that capture multimodal information from human-robot interaction dialogues. To address this gap, we have recorded a novel multimodal dataset (MERCI) that encompasses rich embodied interaction data. The process involved asking participants to complete a questionnaire and gathering their profiles on ten topics, such as hobbies and favorite music. Subsequently, we initiated conversations between the robot and the participants, leveraging GPT-4 to generate contextually appropriate responses based on the participant's profile and emotional state, as determined by facial expression recognition and sentiment analysis. Automatic and user evaluations were conducted to assess the overall quality of the collected data. The results of both evaluations indicated a high level of naturalness, engagement, fluency, consistency, and relevance in the conversation, as well as the robot's ability to provide empathetic responses. It is worth noting that the dataset is derived from genuine interactions with the robot, involving participants who provided personal information and conveyed actual emotions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。