用大模型将机器人真实操作转为自然语言，提升透明度与故障恢复效率
I Can Tell What I am Doing: Toward Real-World Natural Language Grounding of Robot Experiences
- 基于大模型融合多模态数据生成可读叙述
- 在多种场景下提升故障恢复效率，优于现有方法
- 适合关注机器人透明性与人机协作的研究者
通过自然语言理解机器人行为与经验，是构建智能透明机器人系统的关键。尽管大语言模型（LLMs）使复杂多模态机器人经验转化为连贯的人类可读叙述成为可能，但将真实世界机器人经验准确映射到自然语言仍面临挑战，如数据多模态性、采样率差异和数据量大等问题。本文提出RONAR——一个基于大模型的系统，能从机器人经验中生成自然语言叙述，用于行为说明、故障分析及人机交互以实现故障恢复。在多种场景下的评估表明，RONAR优于当前最优方法，显著提升故障恢复效率。贡献包括：一个用于机器人经验叙述的多模态框架、一个全面的真实机器人数据集，以及实证证据证明其在提升系统透明度和故障分析用户体验方面的有效性。
原文摘要 · Abstract (English)
Understanding robot behaviors and experiences through natural language is crucial for developing intelligent and transparent robotic systems. Recent advancement in large language models (LLMs) makes it possible to translate complex, multi-modal robotic experiences into coherent, human-readable narratives. However, grounding real-world robot experiences into natural language is challenging due to many reasons, such as multi-modal nature of data, differing sample rates, and data volume. We introduce RONAR, an LLM-based system that generates natural language narrations from robot experiences, aiding in behavior announcement, failure analysis, and human interaction to recover failure. Evaluated across various scenarios, RONAR outperforms state-of-the-art methods and improves failure recovery efficiency. Our contributions include a multi-modal framework for robot experience narration, a comprehensive real-robot dataset, and empirical evidence of RONAR's effectiveness in enhancing user experience in system transparency and failure analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。