无需机器人,用可穿戴设备高效采集触觉视觉数据
FreeTacMan: Robot-free Visuo-Tactile Data Collection System for Contact-rich Manipulation
- 可穿戴手套配传感器,人手直接操作采集数据
- 收集超300万组视觉触觉图像和10万条操作轨迹
- 适合研究触觉机器人操作的学者与工程师
让机器人掌握丰富的接触式操作仍是机器人学习中的关键挑战,主要受限于数据收集效率低和传感器配置有限。以往的手持方案采用刚性杆状结构,操控不自然且触觉反馈差。受人类动作灵巧性和力反馈启发,我们提出FreeTacMan——一种以人为中心、无需机器人的数据采集系统,实现高精度、高效的机器人操作数据收集。具体而言,设计了一种可佩戴于手指的集成视觉-触觉传感器手套,支持直观的人手控制;引入高精度光学追踪系统,同步捕捉末端执行器位姿与视觉、触觉信号。利用FreeTacMan构建了一个大规模多模态数据集,包含超过300万对视觉-触觉图像及末端位姿,覆盖50种多样化的接触丰富操作任务,共10万条示范轨迹。相比已有方法,FreeTacMan在数据采集性能上实现多项提升,并支持基于自采集数据集的有效策略学习。通过开源硬件与数据集,旨在促进可复现性,推动视觉-触觉操作研究发展。
原文摘要 · Abstract (English)
Enabling robots with contact-rich manipulation remains a pivotal challenge in robot learning, which is substantially hindered by the data collection gap, including its inefficiency and limited sensor setup. While prior work has explored handheld paradigms, their rod-based mechanical structures remain rigid and unintuitive, providing limited tactile feedback and posing challenges for operators. Motivated by the dexterity and force feedback of human motion, we propose FreeTacMan, a human-centric and robot-free data collection system for accurate and efficient robot manipulation. Concretely, we design a wearable gripper with visuo-tactile sensors for data collection, which can be worn by human fingers for intuitive control. A high-precision optical tracking system is introduced to capture end-effector poses while synchronizing visual and tactile feedback simultaneously. We leverage FreeTacMan to collect a large-scale multimodal dataset, comprising over 3000k paired visuo-tactile images with end-effector poses, 10k demonstration trajectories across 50 diverse contact-rich manipulation tasks. FreeTacMan achieves multiple improvements in data collection performance over prior works and enables effective policy learning from self-collected datasets. By open-sourcing the hardware and the dataset, we aim to facilitate reproducibility and support research in visuo-tactile manipulation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。