用语音控制虚拟机器人，无需手柄就能远程操作。
LLM-Driven Augmented Reality Puppeteer: Controller-Free Voice-Commanded Robot Teleoperation
- 通过大模型理解语音指令，实时操控AR中的虚拟机器人
- 在Meta Quest 3上实现免手柄操控，降低操作门槛
- 适合需要安全、直观人机交互的场景，如远程运维
机器人与增强现实（AR）的融合为提升人机交互（HRI）的可用性、直观性和可访问性带来了变革性机遇。本文提出一种无需控制器、由大语言模型（LLM）驱动的语音指令式AR木偶操控系统，允许用户通过实时操控机器人的虚拟副本来远程操作实体机器人。借助自然语言处理（NLP）和AR技术，该系统——基于Meta Quest 3原型实现——消除了对物理控制器的需求，提升了易用性，同时降低了直接操作机器人带来的潜在安全风险。初步用户演示成功验证了系统的功能，展示了其在更安全、更直观、更具沉浸感的机器人控制方面的潜力。
原文摘要 · Abstract (English)
The integration of robotics and augmented reality (AR) presents transformative opportunities for advancing human-robot interaction (HRI) by improving usability, intuitiveness, and accessibility. This work introduces a controller-free, LLM-driven voice-commanded AR puppeteering system, enabling users to teleoperate a robot by manipulating its virtual counterpart in real time. By leveraging natural language processing (NLP) and AR technologies, our system -- prototyped using Meta Quest 3 -- eliminates the need for physical controllers, enhancing ease of use while minimizing potential safety risks associated with direct robot operation. A preliminary user demonstration successfully validated the system's functionality, demonstrating its potential for safer, more intuitive, and immersive robotic control.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。