用分治式扩散模型+路由机制,让机器人更智能地融合视觉触觉信息。
Multi-Modal Manipulation via Multi-Modal Policy Consensus
- 将策略拆成单一模态专用的扩散模型,由路由网络动态加权融合
- 在仿真和真实任务中均显著优于传统拼接特征的方法,尤其在多模态推理场景
- 支持新模态增量加入,且对传感器故障有强鲁棒性,适合复杂交互任务
有效整合多样感官模态对机器人操作至关重要。然而,典型的特征拼接方法常不理想:在接触密集型任务中,视觉等主导模态可能压制稀疏但关键的触觉信号,且单体架构无法灵活加入或缺失模态而不需重新训练。本文将策略分解为一组针对单一表示(如视觉或触觉)的扩散模型,并引入一个学习共识权重的路由器网络,自适应融合各模型贡献,实现新表示的增量集成。我们在RLBench仿真任务及真实世界任务(如遮挡物体抓取、勺子在手旋转、拼图插入)上评估该方法,结果表明其在需要多模态推理的场景中显著优于特征拼接基线。策略还展现出对物理扰动和传感器损坏的鲁棒性。通过扰动重要性分析,揭示了模态间可适应的动态切换机制。
原文摘要 · Abstract (English)
Effectively integrating diverse sensory modalities is crucial for robotic manipulation. However, the typical approach of feature concatenation is often suboptimal: dominant modalities such as vision can overwhelm sparse but critical signals like touch in contact-rich tasks, and monolithic architectures cannot flexibly incorporate new or missing modalities without retraining. Our method factorizes the policy into a set of diffusion models, each specialized for a single representation (e.g., vision or touch), and employs a router network that learns consensus weights to adaptively combine their contributions, enabling incremental of new representations. We evaluate our approach on simulated manipulation tasks in {RLBench}, as well as real-world tasks such as occluded object picking, in-hand spoon reorientation, and puzzle insertion, where it significantly outperforms feature-concatenation baselines on scenarios requiring multimodal reasoning. Our policy further demonstrates robustness to physical perturbations and sensor corruption. We further conduct perturbation-based importance analysis, which reveals adaptive shifts between modalities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。