用眼神和语音驱动的机器人交互系统,提升协作任务中的双向互动能力。
Gaze-supported Large Language Model Framework for Bi-directional Human-Robot Interaction
- 结合眼动与语音输入,实现多模态环境感知与动态交互
- 相比传统脚本系统,适应性更强但存在冗余输出问题
- 适用于需要灵活响应的辅助机器人场景
大型语言模型(LLMs)的快速发展为通用知识驱动的辅助机器人人机交互系统带来了巨大潜力。现有系统在理解用户指令、生成动作和解决任务方面已取得显著进展,但在协作任务中实现双向、多模态和上下文感知的用户支持仍面临挑战。本文提出一种基于眼动与语音信息的交互界面,使辅助机器人能够从多个视觉输入中感知工作环境,并动态支持用户完成任务。系统设计模块化且可迁移,适用于不同任务和机器人平台,具备实时语言交互状态表示与快速本地感知能力。开发过程中通过多次公开活动验证,提升了鲁棒性和用户体验。在两项实验室研究中,我们对比了该系统与传统脚本化人机交互流程的表现与用户评价。结果表明,基于LLM的方法提升了适应性,略微改善了用户参与度与任务执行指标,但可能产生冗余输出;而脚本化流程更适合简单明确的任务。
原文摘要 · Abstract (English)
The rapid development of Large Language Models (LLMs) creates an exciting potential for flexible, general knowledge-driven Human-Robot Interaction (HRI) systems for assistive robots. Existing HRI systems demonstrate great progress in interpreting and following user instructions, action generation, and robot task solving. On the other hand, bi-directional, multi-modal, and context-aware support of the user in collaborative tasks still remains an open challenge. In this paper, we present a gaze- and speech-informed interface to the assistive robot, which is able to perceive the working environment from multiple vision inputs and support the dynamic user in their tasks. Our system is designed to be modular and transferable to adapt to diverse tasks and robots, and it is capable of real-time use of language-based interaction state representation and fast on board perception modules. Its development was supported by multiple public dissemination events, contributing important considerations for improved robustness and user experience. Furthermore, in two lab studies, we compare the performance and user ratings of our system with those of a traditional scripted HRI pipeline. Our findings indicate that an LLM-based approach enhances adaptability and marginally improves user engagement and task execution metrics but may produce redundant output, while a scripted pipeline is well suited for more straightforward tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。