构建了包含视觉触觉动作的多模态机器人操作数据集,助力类人操控学习。
RoboTacDex: A Dexterous Visual-Tactile-Action Dataset for Humanoid Manipulation
- 基于双臂灵巧手机器人采集6000条轨迹,融合多视角图像、深度与触觉信号
- 涵盖19项任务、23种技能、22类物体,覆盖复杂协同操作场景
- 适合作为仿人操作学习、多模态强化学习的研究基准
在机器人学习领域,大规模多样化的示范轨迹是提升机器人操作能力的基础。我们提出了RoboTacDex,一个大规模、多模态、多样化的类人机器人灵巧操作数据集。该数据集基于公开可用的人形机器人Unitree G1构建,包含6000条轨迹,覆盖19个任务、23种技能和22类物体的交互。数据集提供多视角RGB与深度信息、触觉反馈以及详细的语义标注。此外,数据集包含多种需双臂与灵巧手协同完成的高难度任务,旨在模拟人类操作逻辑并还原真实世界操作复杂性。为保证数据质量,我们设计改进的多相机同步系统,实现毫秒级模态同步记录。实验中,我们在数据集上评估了三种代表性模仿学习模型,分析其在不同任务类别中的表现及优缺点。成功试验结果与中等水平的跨任务泛化能力表明所收集数据集的有效性与多样性。本数据集即将开源。
原文摘要 · Abstract (English)
In the field of robot learning, large-scale and diverse demonstration trajectories provide the fundamental basis for enhancing robotic manipulation ability. We introduce RoboTacDex, a large, multi-modal, and diverse dataset of dexterous manipulation behaviors performed with a humanoid robot. Built on the publicly accessible humanoid robot Unitree G1, RoboTacDex consists of 6k trajectories covering 19 tasks, 23 skills, and interactions with 22 objects. RoboTacDex provides comprehensive records including multi-view RGB and depth information, tactile feedback, and detailed semantic annotations. Furthermore, the dataset features a variety of relatively challenging tasks that can only be completed by dual arms and dexterous hands, aiming to mimic human-like operational logic and simulate real-world manipulation complexity. To ensure data collection quality, we develop an improved multi-camera synchronization system to enable millisecond data synchronization and recording of modalities. In our experiments, we evaluate three representative imitation learning models on our dataset, analyzing their performance as well as their respective strengths and limitations across different task categories. Successful trial results and a moderate level of generalization capabilities across a suite of tasks indicate the effectiveness and diversity of the collected dataset. Our dataset will be open-sourced soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。