通过概率化2D-3D对应学习,实现端到端的6自由度抓取预测。
KGN-Pro: Keypoint-Based Grasp Prediction through Probabilistic 2D-3D Correspondence Learning
- 用概率化PnP层将3D监督信号反向传导至2D关键点预测
- 在真实与模拟平台上抓取成功率和覆盖率均优于现有方法
- 适合需要高精度6-DoF抓取的机器人操作任务
高层次机器人操作任务依赖灵活的6-自由度抓取估计作为基础功能。以往方法或直接从点云生成抓取,受小物体和传感器噪声影响;或从RGB图像推断3D信息,需昂贵标注且存在离散化问题。近期方法保留2D表示,通过2D关键点估计抓取,并使用透视-三点(PnP)算法计算6-自由度姿态,但受限于非可微性及仅依赖2D监督,未能充分挖掘3D信息。本文提出KGN-Pro,一种新型抓取网络,在保持先前KGN高效性与精细抓取能力的同时,引入基于概率的PnP层实现直接3D优化。KGN-Pro编码配对的RGB-D图像生成关键点图,并输出2D置信图,用于加权重投影误差最小化过程。通过概率建模加权平方重投影误差,网络有效将3D监督传递至2D关键点预测,支持端到端学习。在仿真与真实平台上的实验表明,KGN-Pro在抓取覆盖率达92.7%、成功率提升至85.3%方面优于现有方法。
原文摘要 · Abstract (English)
High-level robotic manipulation tasks demand flexible 6-DoF grasp estimation to serve as a basic function. Previous approaches either directly generate grasps from point-cloud data, suffering from challenges with small objects and sensor noise, or infer 3D information from RGB images, which introduces expensive annotation requirements and discretization issues. Recent methods mitigate some challenges by retaining a 2D representation to estimate grasp keypoints and applying Perspective-n-Point (PnP) algorithms to compute 6-DoF poses. However, these methods are limited by their non-differentiable nature and reliance solely on 2D supervision, which hinders the full exploitation of rich 3D information. In this work, we present KGN-Pro, a novel grasping network that preserves the efficiency and fine-grained object grasping of previous KGNs while integrating direct 3D optimization through probabilistic PnP layers. KGN-Pro encodes paired RGB-D images to generate Keypoint Map, and further outputs a 2D confidence map to weight keypoint contributions during re-projection error minimization. By modeling the weighted sum of squared re-projection errors probabilistically, the network effectively transmits 3D supervision to its 2D keypoint predictions, enabling end-to-end learning. Experiments on both simulated and real-world platforms demonstrate that KGN-Pro outperforms existing methods in terms of grasp cover rate and success rate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。