仅用一张图的框标注,让机器人学会在千变万化场景中操作。
RoboAug: One Annotation to Hundreds of Scenes via Region-Contrastive Data Augmentation for Robotic Manipulation
- 用单张图像框标注+生成模型,实现精准语义数据增强。
- 三台机器人35000次实测,成功率从不足20%提升至67%。
- 无需完美识别,适合真实复杂环境下的机器人训练。
提升机器人学习在多样未见场景中的泛化能力是核心挑战。现有方法多依赖大规模预训练数据或理想化的语义增强假设,前者耗时费力,后者在现实场景中难以满足。本文提出RoboAug,一种新型生成式数据增强框架,仅需单张图像的边界框标注即可训练,大幅降低对大规模预训练和完美视觉识别的依赖。该方法利用预训练生成模型进行精确语义增强,并引入即插即用的区域对比损失,引导模型关注任务相关区域,从而提升泛化性能。我们在三台机器人(UR-5e、AgileX、Tien Kung 2.0)上开展大规模实测,共超过35,000次轨迹评估。结果表明,面对不同背景、干扰物和光照组合的未见场景,该方法显著优于主流基线:在UR-5e上成功率从0.09升至0.47,在AgileX上从0.16升至0.60,在Tien Kung 2.0上从0.19升至0.67,充分验证其在真实操作任务中的卓越泛化性与有效性。
原文摘要 · Abstract (English)
Enhancing the generalization capability of robotic learning to enable robots to operate effectively in diverse, unseen scenes is a fundamental and challenging problem. Existing approaches often depend on pretraining with large-scale data collection, which is labor-intensive and time-consuming, or on semantic data augmentation techniques that necessitate an impractical assumption of flawless upstream object detection in real-world scenarios. In this work, we propose RoboAug, a novel generative data augmentation framework that significantly minimizes the reliance on large-scale pretraining and the perfect visual recognition assumption by requiring only the bounding box annotation of a single image during training. Leveraging this minimal information, RoboAug employs pre-trained generative models for precise semantic data augmentation and integrates a plug-and-play region-contrastive loss to help models focus on task-relevant regions, thereby improving generalization and boosting task success rates. We conduct extensive real-world experiments on three robots, namely UR-5e, AgileX, and Tien Kung 2.0, spanning over 35k rollouts. Empirical results demonstrate that RoboAug significantly outperforms state-of-the-art data augmentation baselines. Specifically, when evaluating generalization capabilities in unseen scenes featuring diverse combinations of backgrounds, distractors, and lighting conditions, our method achieves substantial gains over the baseline without augmentation. The success rates increase from 0.09 to 0.47 on UR-5e, from 0.16 to 0.60 on AgileX, and from 0.19 to 0.67 on Tien Kung 2.0. These results highlight the superior generalization and effectiveness of RoboAug in real-world manipulation tasks. Our project is available at https://x-roboaug.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。