让机器人在杂乱场景中零样本实机抓取物体,成功率超89%。
FetchBot: Learning Generalizable Object Fetching in Cluttered Scenes via Zero-Shot Sim2Real
- 用100万场景的仿真数据训练,通过深度图和结构线索规划避障动作。
- 真实环境中抓取成功率高达89.95%,优于已有方法。
- 可处理透明、反光等难物,适合工业自动化场景部署。
在杂乱场景中实现泛化抓取仍是具身智能的核心挑战。密集物体导致严重遮挡,使安全动作生成极为困难。在此部分可观测条件下,有效策略需在不同物体与布局间泛化,并推理遮挡以避免碰撞。然而,大规模真实数据采集成本过高,制约该问题解决。本文提出FetchBot,一种面向此任务的零样本仿真到现实迁移框架。我们构建了一个包含100万种多样化场景与50万条示范动作的合成数据集。基于此,FetchBot采用深度条件动作生成方法,利用结构线索实现鲁棒的障碍感知规划。由于真实环境深度数据噪声大,我们引入基于基础模型的RGB转深度预测,并将局部占据预测作为预训练任务,获得可迁移的隐表示。仿真与真实环境中的大量实验表明,该方法具备强零样本仿真到现实迁移能力,能有效处理杂乱场景,并适应新情境。在复杂环境中,平均真实成功率达89.95%,显著优于先前方法。此外,其在透明、反射及不规则物体上的表现优异,凸显实际应用价值。
原文摘要 · Abstract (English)
Generalizable object fetching in cluttered scenes remains a fundamental and application-critical challenge in embodied AI. Closely packed objects cause inevitable occlusions, making safe action generation particularly difficult. Under such partial observability, effective policies must not only generalize across diverse objects and layouts but also reason about occlusion to avoid collisions. However, collecting large-scale real-world data for this task remains prohibitively expensive, leaving this problem largely unsolved. In this paper, we introduce FetchBot, a sim-to-real framework for this challenge. We first curate a large-scale synthetic dataset featuring 1M diverse scenes and 500k representative demonstrations. Based on this dataset, FetchBot employs a depth-conditioned method for action generation, which leverages structural cues to enable robust obstacle-aware action planning. However, depth is perfect in simulation but noisy in real-world environments. To address this sim-to-real gap, FetchBot predicts depth from RGB inputs using a foundation model and integrates local occupancy prediction as a pre-training task, providing a generalizable latent representation for sim-to-real transfer. Extensive experiments in simulation and real-world environments demonstrate the strong zero-shot sim-to-real transfer, effective clutter handling, and adaptability to novel scenarios. In cluttered environments, it achieves an average real-world success rate of 89.95%, significantly outperforming prior methods. Moreover, FetchBot demonstrates excellent robustness in challenging cases, such as fetching transparent, reflective, and irregular objects, highlighting its practical value.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。