让机器人从乱堆的衣物中精准取出单件衣服,支持语言指令和复杂抓取。
GarmentPile++: Affordance-Driven Cluttered Garments Retrieval with Vision-Language Reasoning
- 结合视觉语言模型与物体可操作性感知,实现多任务推理。
- 在真实与仿真环境中均能准确取出单件衣物,成功率高。
- 适合家庭服务机器人、智能仓储等需要精细操作的场景。
衣物操作因在家庭助手机器人中的关键作用而受到越来越多关注。然而,现有研究大多假设初始状态仅有一件衣物,而现实中衣物堆叠更为常见。为此,我们提出一种新型衣物检索流程,不仅能根据语言指令安全、干净地完成检索,还能确保每次尝试恰好取出一件衣物,为后续任务(如折叠、悬挂、穿戴)提供可靠基础。该流程无缝融合视觉-语言推理与视觉可操作性感知,充分利用视觉语言模型(VLM)的高层推理与规划能力,以及视觉可操作性的低层动作泛化能力。为增强VLM对堆叠衣物中每件物品状态的全面理解,我们采用视觉分割模型(SAM2)对衣物堆进行对象分割,为VLM推理提供充分视觉线索;同时引入掩码微调机制,应对初始分割效果不佳的情况。此外,部署双臂协作框架,以处理大件或长条形衣物,以及因抓取点选择错误导致的过度下垂问题,这些对单臂操作而言极具挑战。该流程在真实世界与仿真环境中多种任务和场景下均表现出色。
原文摘要 · Abstract (English)
Garment manipulation has attracted increasing attention due to its critical role in home-assistant robotics. However, the majority of existing garment manipulation works assume an initial state consisting of only one garment, while piled garments are far more common in real-world settings. To bridge this gap, we propose a novel garment retrieval pipeline that can not only follow language instruction to execute safe and clean retrieval but also guarantee exactly one garment is retrieved per attempt, establishing a robust foundation for the execution of downstream tasks (e.g., folding, hanging, wearing). Our pipeline seamlessly integrates vision-language reasoning with visual affordance perception, fully leveraging the high-level reasoning and planning capabilities of VLMs alongside the generalization power of visual affordance for low-level actions. To enhance the VLM's comprehensive awareness of each garment's state within a garment pile, we employ visual segmentation model (SAM2) to execute object segmentation on the garment pile for aiding VLM-based reasoning with sufficient visual cues. A mask fine-tuning mechanism is further integrated to address scenarios where the initial segmentation results are suboptimal. In addition, a dual-arm cooperation framework is deployed to address cases involving large or long garments, as well as excessive garment sagging caused by incorrect grasping point determination, both of which are strenuous for a single arm to handle. The effectiveness of our pipeline are consistently demonstrated across diverse tasks and varying scenarios in both real-world and simulation environments. Project page: https://garmentpile2.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。