融合多视角点云与手部高清视频,提升远程机器人操作的沉浸感与精准度。
A Multi-View 3D Telepresence System for XR Robot Teleoperation
- 三相机融合生成点云,结合腕部摄像头提供局部高清细节。
- 系统实现在Meta Quest 3上实时渲染约7.5万点,支持低延迟交互。
- 适合需要高精度远程操控的救援、维护等场景,对用户体验提升明显。
机器人远程操控在远程维护、车队机器人、搜救及机器人学习数据采集中至关重要。有效操控依赖直观的三维可视化和可靠的深度线索,而传统屏幕界面常难以满足。本文提出一种多视角VR远程存在系统:(1) 通过三台相机融合几何信息,在独立式VR硬件上实现GPU加速的点云渲染;(2) 集成腕部佩戴的RGB流,在点云精度受限区域提供高分辨率局部细节。该流水线可在Meta Quest 3上实时渲染约7.5万点。通过31名参与者参与的组内研究,对比了本系统与其他可视化方式(如纯RGB流、直接投影立体视觉至VR设备、无额外RGB信息的点云)的表现。在三项不同遥控操作任务中,评估了任务成功率、完成时间、主观工作负荷与可用性。结果表明,本系统整体表现最优,且无额外RGB信息的点云模态也优于纯RGB流与OpenTeleVision。说明将全局三维结构与局部高分辨率细节结合,显著提升操作沉浸感,为下一代机器人远程操控系统奠定坚实基础。
原文摘要 · Abstract (English)
Robot teleoperation is critical for applications such as remote maintenance, fleet robotics, search and rescue, and data collection for robot learning. Effective teleoperation requires intuitive 3D visualization with reliable depth cues, which conventional screen-based interfaces often fail to provide. We introduce a multi-view VR telepresence system that (1) fuses geometry from three cameras to produce GPU-accelerated point-cloud rendering on standalone VR hardware, and (2) integrates a wrist-mounted RGB stream to provide high-resolution local detail where point-cloud accuracy is limited. Our pipeline supports real-time rendering of approximately 75k points on the Meta Quest 3. A within-subject study was conducted with 31 participants to compare our system to other visualisation modalities, such as RGB streams, a projection of stereo-vision directly in the VR device and point clouds without providing additional RGB information. Across three different teleoperated manipulation tasks, we measured task success, completion time, perceived workload, and usability. Our system achieved the best overall performance, while the Point Cloud modality without RGB also outperforming the RGB streams and OpenTeleVision. These results show that combining global 3D structure with localized high-resolution detail substantially improves telepresence for manipulation and provides a strong foundation for next-generation robot teleoperation systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。