用低成本设备采集海量真实场景触觉数据，提升机器人材料识别能力。
CLAMP: Crowdsourcing a LArge-scale in-the-wild haptic dataset with an open-source device for Multimodal robot Perception
- 设计低成本触觉抓取器，让普通人日常使用中收集触觉数据。
- 建成1230万条数据的开源触觉数据库，覆盖5357种家庭物品。
- 模型可跨机器人形态泛化，在垃圾分类等任务中表现优异。
在非结构化环境中实现鲁棒机器人操作常需理解物体超出几何属性的特性，如材质或柔顺性，这些仅靠视觉难以推断。多模态触觉感知为此提供可行路径，但受限于缺乏大规模、多样化且真实的触觉数据集。本文提出CLAMP设备——一种成本低于200美元的传感器化抓取装置，用于在日常环境中由非专家用户收集大规模真实场景下的多模态触觉数据。我们向41名参与者部署了16台CLAMP设备，构建了目前最大的开源多模态触觉数据集：CLAMP数据集，包含1230万条数据点，覆盖5357种家用物品。基于该数据集，我们训练了一个触觉编码器，可从多模态触觉信号中推断物体材质与柔顺性。进一步构建了CLAMP模型，一个具有强泛化能力的视觉-触觉感知模型，适用于新物体及三种不同机器人形态，仅需少量微调。我们在三个真实机器人任务中验证其有效性：分类可回收与不可回收垃圾、从杂乱袋中取出物品、区分过熟与成熟香蕉。结果表明，大规模真实场景触觉数据采集能显著提升机器人通用操作能力。
原文摘要 · Abstract (English)
Robust robot manipulation in unstructured environments often requires understanding object properties that extend beyond geometry, such as material or compliance-properties that can be challenging to infer using vision alone. Multimodal haptic sensing provides a promising avenue for inferring such properties, yet progress has been constrained by the lack of large, diverse, and realistic haptic datasets. In this work, we introduce the CLAMP device, a low-cost (<\$200) sensorized reacher-grabber designed to collect large-scale, in-the-wild multimodal haptic data from non-expert users in everyday settings. We deployed 16 CLAMP devices to 41 participants, resulting in the CLAMP dataset, the largest open-source multimodal haptic dataset to date, comprising 12.3 million datapoints across 5357 household objects. Using this dataset, we train a haptic encoder that can infer material and compliance object properties from multimodal haptic data. We leverage this encoder to create the CLAMP model, a visuo-haptic perception model for material recognition that generalizes to novel objects and three robot embodiments with minimal finetuning. We also demonstrate the effectiveness of our model in three real-world robot manipulation tasks: sorting recyclable and non-recyclable waste, retrieving objects from a cluttered bag, and distinguishing overripe from ripe bananas. Our results show that large-scale, in-the-wild haptic data collection can unlock new capabilities for generalizable robot manipulation. Website: https://emprise.cs.cornell.edu/clamp/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。