融合RGB与深度信息提升视觉强化学习的仿真到现实迁移能力
Multimodal Fusion for Sim2real Transfer in Visual Reinforcement Learning
- 用Vision Transformer融合RGB与深度图像特征
- 在多个环境中实现零样本迁移，真实任务成功率达78%
- 适合做机器人操控等需要跨域泛化的研究者
深度信息对场景外观变化具有鲁棒性，并天然包含三维空间细节。本文提出基于视觉变换器的多模态融合方法，将RGB与深度图结合以增强模型泛化能力。不同模态先通过独立的CNN主干处理，再将融合后的卷积特征输入可扩展的视觉变换器生成视觉表征。同时设计了掩码与非掩码令牌的对比学习策略，提升样本效率与泛化性能。采用基于课程的领域随机化方案，灵活稳定训练过程。仿真结果表明，该融合方案优于多个基线模型。通过零样本迁移，在真实世界操作任务中验证了模型可行性。
原文摘要 · Abstract (English)
Depth information is robust to scene appearance variations and inherently carries 3D spatial details. Thus, a visual backbone based on the vision transformer is proposed to fuse RGB and depth modalities for enhancing generalization in this paper. Different modalities are first processed by separate CNN stems, and the combined convolutional features are delivered to the scalable vision transformer to obtain visual representations. Moreover, a contrastive learning scheme is designed with masked and unmasked tokens to enhance the sample efficiency and generalization performance. A curriculum-based domain randomization scheme is used to flexibly stabilize the training process. Finally, simulation results demonstrate that our fusion scheme outperforms the other baselines. The feasibility of our model is validated to perform real-world manipulation tasks via zero-shot transfer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。