用视觉检测+文本提示实现机器人通用抓取，成功率超90%。
Versatile and Generalizable Manipulation via Goal-Conditioned Reinforcement Learning with Grounded Object Detection
- 用预训练检测模型将文本指令转为物体掩码，指导机器人抓取。
- 在分布外物体上仍保持约90%抓取成功率，收敛更快。
- 适合需要快速适配新任务的机器人应用场景。
通用机器人操作（如抓取与移动）对家庭和工作环境部署至关重要。近期研究利用大模型（如大语言模型和物体检测器）提升强化学习中的机器人感知能力。这些模型通过自监督学习在大规模数据上训练，可处理文本指令并识别场景中多样物体，有效缓解强化学习中物体交互学习的资源消耗问题。本文提出将预训练物体检测模型融入目标条件强化学习，实现通用抓取能力。通过检测模型将文本提示映射为物体掩码，用于目标条件输入；掩码作为目标条件能提供与物体无关的线索，增强特征共享与泛化能力。在模拟抓取任务中，该方法在分布内与分布外物体上均维持约90%的成功率，并加速达到高回报策略。
原文摘要 · Abstract (English)
General-purpose robotic manipulation, including reach and grasp, is essential for deployment into households and workspaces involving diverse and evolving tasks. Recent advances propose using large pre-trained models, such as Large Language Models and object detectors, to boost robotic perception in reinforcement learning. These models, trained on large datasets via self-supervised learning, can process text prompts and identify diverse objects in scenes, an invaluable skill in RL where learning object interaction is resource-intensive. This study demonstrates how to integrate such models into Goal-Conditioned Reinforcement Learning to enable general and versatile robotic reach and grasp capabilities. We use a pre-trained object detection model to enable the agent to identify the object from a text prompt and generate a mask for goal conditioning. Mask-based goal conditioning provides object-agnostic cues, improving feature sharing and generalization. The effectiveness of the proposed framework is demonstrated in a simulated reach-and-grasp task, where the mask-based goal conditioning consistently maintains a $\sim$90\% success rate in grasping both in and out-of-distribution objects, while also ensuring faster convergence to higher returns.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。