用虚拟视角提升机器人3D操作效率，训练快1.89倍，推理快1.54倍。
VERM: Leveraging Foundation Models to Create a Virtual Eye for Efficient 3D Robotic Manipulation
- 利用基础模型生成任务自适应的虚拟视角，减少多摄像头冗余信息。
- 在RLBench仿真与真实场景中均超越现有方法，训练提速1.89倍，推理提速1.54倍。
- 适合需要高效3D操作的机器人系统，尤其关注计算效率与精度平衡的场景。
执行3D操作任务时，机器人需基于多个固定摄像头的感知进行动作规划。多摄像头设置引入大量冗余和无关信息，增加计算开销，并迫使模型耗费额外训练时间提取关键任务特征。为过滤冗余信息并精准提取任务相关特征，我们提出VERM（Virtual Eye for Robotic Manipulation）方法，借助基础模型知识，从构建的3D点云中生成虚拟任务自适应视图，有效捕捉必要信息并缓解遮挡问题。为进一步支持3D动作规划与精细操作，我们设计了深度感知模块和动态粗到精流程。在仿真基准RLBench及真实世界评估中，实验结果证明该方法有效性，优于先前最先进方法，同时实现训练时间1.89倍加速与推理速度1.54倍加速。更多结果见项目网站：https://verm-ral.github.io。
原文摘要 · Abstract (English)
When performing 3D manipulation tasks, robots have to execute action planning based on perceptions from multiple fixed cameras. The multi-camera setup introduces substantial redundancy and irrelevant information, which increases computational costs and forces the model to spend extra training time extracting crucial task-relevant details. To filter out redundant information and accurately extract task-relevant features, we propose the VERM (Virtual Eye for Robotic Manipulation) method, leveraging the knowledge in foundation models to imagine a virtual task-adaptive view from the constructed 3D point cloud, which efficiently captures necessary information and mitigates occlusion. To facilitate 3D action planning and fine-grained manipulation, we further design a depth-aware module and a dynamic coarse-to-fine procedure. Extensive experimental results on both simulation benchmark RLBench and real-world evaluations demonstrate the effectiveness of our method, surpassing previous state-of-the-art methods while achieving 1.89x speedup in training time and 1.54x speedup in inference speed. More results can be found on our project website at https://verm-ral.github.io .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。