让机器人在空中抓握皱衣服,靠视觉触觉结合智能判断何时该动。
Reactive In-Air Clothing Manipulation with Confidence-Aware Dense Correspondence and Visuotactile Affordance
- 用视觉+触觉融合判断衣物是否可抓,动态调整操作策略。
- 在低置信度时暂停动作,成功处理高度遮挡的皱衣和悬空状态。
- 适合需要灵活抓取复杂布料的机器人应用,如智能熨烫或衣柜整理。
由于衣物结构复杂、材质动态变化频繁且易自遮挡,操控衣物极具挑战。现有系统常将衣物展平或假设关键特征可见。本文提出双臂视觉触觉框架,结合置信度感知的密集视觉对应关系与触觉监督的抓取可操作性,直接对褶皱和悬空衣物进行操作。对应模型在自建高保真仿真数据集上训练,采用分布损失捕捉布料对称性,并生成对应置信度估计;这些估计指导反应式状态机,根据感知不确定性动态调整折叠策略。同时,基于高分辨率触觉反馈自监督训练的视觉触觉抓取可操作性网络,判定哪些区域可物理抓取,执行中亦用于实时抓取验证。通过在置信度低时延迟动作,系统能有效应对高度遮挡的桌面及空中配置。我们验证了任务无关的抓取选择模块在折叠与悬挂任务中的有效性。此外,密集描述符可作为其他规划模态的通用中间表示,例如从人类视频演示中提取抓取目标,为更通用、可扩展的衣物操作铺路。
原文摘要 · Abstract (English)
Manipulating clothing is challenging due to complex configurations, variable material dynamics, and frequent self-occlusion. Prior systems often flatten garments or assume visibility of key features. We present a dual-arm visuotactile framework that combines confidence-aware dense visual correspondence and tactile-supervised grasp affordance to operate directly on crumpled and suspended garments. The correspondence model is trained on a custom, high-fidelity simulated dataset using a distributional loss that captures cloth symmetries and generates correspondence confidence estimates. These estimates guide a reactive state machine that adapts folding strategies based on perceptual uncertainty. In parallel, a visuotactile grasp affordance network, self-supervised using high-resolution tactile feedback, determines which regions are physically graspable. The same tactile classifier is used during execution for real-time grasp validation. By deferring action in low-confidence states, the system handles highly occluded table-top and in-air configurations. We demonstrate our task-agnostic grasp selection module in folding and hanging tasks. Moreover, our dense descriptors provide a reusable intermediate representation for other planning modalities, such as extracting grasp targets from human video demonstrations, paving the way for more generalizable and scalable garment manipulation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。