提出DECO模型，让机器人双手操作更灵巧，触觉信息提升成功率20%。
DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter
- 分路处理视觉、本体和触觉信号，实现多模态精准融合。
- 真实机器人测试中平均成功率达72.25%，比基线高21%。
- 轻量触觉适配器仅调10%参数，触觉任务性能再提升20%。
双臂灵巧操作依赖多模态输入融合完成复杂任务。为解决模态有效结合的挑战，我们提出DECO——一种解耦式多模态扩散变压器，通过专用条件路径分离视觉、本体和触觉信号，实现结构化且可控的多模态融合，并引入轻量级适配器，以参数高效方式注入额外信号。同时，我们发布DECO-50数据集，包含50小时真实双臂机器人遥操作采集的数据，超过500万帧，覆盖多样任务。在DECO-50上训练DECO，并进行超过2000次真实机器人滚动评估。实验表明，DECO在所有任务中表现最优，平均成功率达72.25%，较基线提升21%。此外，触觉适配器在所有任务上带来10.25%的平均成功率提升，在复杂接触密集任务中提升达20%，且仅需调整模型不到10%的参数。
原文摘要 · Abstract (English)
Bimanual dexterous manipulation relies on integrating multimodal inputs to perform complex real-world tasks. To address the challenges of effectively combining these modalities, we propose DECO, a decoupled multimodal diffusion transformer that disentangles vision, proprioception, and tactile signals through specialized conditioning pathways, enabling structured and controllable integration of multimodal inputs, with a lightweight adapter for parameter-efficient injection of additional signals. Alongside DECO, we release DECO-50 dataset for bimanual dexterous manipulation with tactile sensing, consisting of 50 hours of data and over 5M frames, collected via teleoperation on real dual-arm robots. We train DECO on DECO-50 and conduct extensive real-world evaluation with over 2,000 robot rollouts. Experimental results show that DECO achieves the best performance across all tasks, with a 72.25% average success rate and a 21% improvement over the baseline. Moreover, the tactile adapter brings an additional 10.25% average success rate across all tasks and a 20% gain on complex contact-rich tasks while tuning less than 10% of the model parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。