PRISM通过点云分割与注意力融合，实现复杂场景下机器人抓取的精准控制。
PRISM: Pointcloud Reintegrated Inference via Segmentation and Cross-attention for Manipulation
- 基于点云分割与跨注意力机制，融合视觉与关节状态信息
- 仅用100次示范即在模拟环境中超越2D/3D基线方法
- 适合需要高鲁棒性3D感知的机器人操作任务
机器人操作的鲁棒模仿学习依赖全面的3D感知，但现有方法在杂乱环境中表现不佳。固定摄像头视角易受视角变化影响，而3D点云技术常局限于关键帧预测，难以应对动态、接触密集的任务。为此，我们提出PRISM，一个端到端框架，直接从原始点云观测和机器人状态学习，无需预训练模型或外部数据集。PRISM包含三个核心组件：分割嵌入单元将原始点云划分为不同物体簇并编码局部几何特征；跨注意力模块将视觉特征与处理后的机器人关节状态融合，突出目标区域；扩散模块将融合表示转化为平滑机器人动作。在每项任务仅使用100次示范训练的情况下，PRISM在模拟环境中性能优于2D与3D基线策略，展现出在复杂、物体密集场景中的强鲁棒性。代码与部分演示见https://github.com/czknuaa/PRISM。
原文摘要 · Abstract (English)
Robust imitation learning for robot manipulation requires comprehensive 3D perception, yet many existing methods struggle in cluttered environments. Fixed camera view approaches are vulnerable to perspective changes, and 3D point cloud techniques often limit themselves to keyframes predictions, reducing their efficacy in dynamic, contact-intensive tasks. To address these challenges, we propose PRISM, designed as an end-to-end framework that directly learns from raw point cloud observations and robot states, eliminating the need for pretrained models or external datasets. PRISM comprises three main components: a segmentation embedding unit that partitions the raw point cloud into distinct object clusters and encodes local geometric details; a cross-attention component that merges these visual features with processed robot joint states to highlight relevant targets; and a diffusion module that translates the fused representation into smooth robot actions. With training on 100 demonstrations per task, PRISM surpasses both 2D and 3D baseline policies in accuracy and efficiency within our simulated environments, demonstrating strong robustness in complex, object-dense scenarios. Code and some demos are available on https://github.com/czknuaa/PRISM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。