arXiv:2507.13468cs.ROcs.AI2025-07被引 8

用多模态信号检测人机对话中机器人错误,提升交互可靠性。

ERR@HRI 2.0 Challenge: Multimodal Detection of Errors and Failures in Human-Robot Conversations

  • 结合面部、语音和头部动作数据,识别机器人对话故障
  • 涵盖16小时人机对话,标注系统级错误与用户纠正意图
  • 适合研究人机交互、故障检测与多模态机器学习的学者

将大语言模型(LLMs)集成到对话机器人中,使人机对话更加动态。然而,基于LLM的对话机器人仍易出现误解用户意图、过早打断或完全无响应等错误。检测并处理这些故障对防止对话中断、避免任务失败及维持用户信任至关重要。为此,ERR@HRI 2.0挑战赛提供了一个多模态数据集,包含16小时双人互动中的机器人故障记录,涵盖面部、语音和头部运动特征。每段交互均从系统视角标注是否存在机器人错误,并标注用户是否意图纠正行为与期望之间的不匹配。研究者可组队开发基于多模态数据的故障检测模型,通过检测准确率与误报率等指标进行评估。该挑战推动了通过社会信号分析提升人机交互中故障检测能力的进展。

原文摘要 · Abstract (English)

The integration of large language models (LLMs) into conversational robots has made human-robot conversations more dynamic. Yet, LLM-powered conversational robots remain prone to errors, e.g., misunderstanding user intent, prematurely interrupting users, or failing to respond altogether. Detecting and addressing these failures is critical for preventing conversational breakdowns, avoiding task disruptions, and sustaining user trust. To tackle this problem, the ERR@HRI 2.0 Challenge provides a multimodal dataset of LLM-powered conversational robot failures during human-robot conversations and encourages researchers to benchmark machine learning models designed to detect robot failures. The dataset includes 16 hours of dyadic human-robot interactions, incorporating facial, speech, and head movement features. Each interaction is annotated with the presence or absence of robot errors from the system perspective, and perceived user intention to correct for a mismatch between robot behavior and user expectation. Participants are invited to form teams and develop machine learning models that detect these failures using multimodal data. Submissions will be evaluated using various performance metrics, including detection accuracy and false positive rate. This challenge represents another key step toward improving failure detection in human-robot interaction through social signal analysis.

人机交互故障检测多模态分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。