用规则点阵融合视觉与点云，提升机器人抓取的精度和泛化能力。
PointMapPolicy: Structured Point Cloud Processing for Multi-Modal Imitation Learning
- 将点云转为规则网格结构，保留细节并支持直接用视觉算法处理
- 在RoboCasa和CALVIN上达到当前最优性能，真实机器人验证有效
- 结合xLSTM融合多模态信息，适合复杂场景下的机器人操作任务
机器人操作系统受益于多种传感模态的互补性：点云捕捉精细几何结构，而RGB图像提供丰富的语义信息。现有点云方法难以保留细粒度细节，尤其在复杂任务中；而RGB方法缺乏几何感知，影响精度与泛化能力。我们提出PointMapPolicy，一种在非下采样规则点阵上条件化扩散策略的新方法。该结构化数据便于提取形状与空间关系，并可轻松在不同参考帧间转换。由于其规则网格特性，可直接应用成熟的计算机视觉技术处理3D数据。采用xLSTM作为主干网络，高效融合点阵与RGB信息以增强多模态感知。在RoboCasa和CALVIN基准测试及真实机器人评估中，本方法在多样化操作任务中均实现领先性能。项目主页展示概览与演示视频：https://point-map.github.io/Point-Map/
原文摘要 · Abstract (English)
Robotic manipulation systems benefit from complementary sensing modalities, where each provides unique environmental information. Point clouds capture detailed geometric structure, while RGB images provide rich semantic context. Current point cloud methods struggle to capture fine-grained detail, especially for complex tasks, which RGB methods lack geometric awareness, which hinders their precision and generalization. We introduce PointMapPolicy, a novel approach that conditions diffusion policies on structured grids of points without downsampling. The resulting data type makes it easier to extract shape and spatial relationships from observations, and can be transformed between reference frames. Yet due to their structure in a regular grid, we enable the use of established computer vision techniques directly to 3D data. Using xLSTM as a backbone, our model efficiently fuses the point maps with RGB data for enhanced multi-modal perception. Through extensive experiments on the RoboCasa and CALVIN benchmarks and real robot evaluations, we demonstrate that our method achieves state-of-the-art performance across diverse manipulation tasks. The overview and demos are available on our project page: https://point-map.github.io/Point-Map/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。