用对比学习对齐视觉触觉特征，提升机器人感知精度。
ConViTac: Aligning Visual-Tactile Fusion with Contrastive Representations
- 引入对比嵌入条件机制，统一视觉与触觉的特征空间。
- 在材料分类和抓取预测任务中最高提升12.0%准确率。
- 适合做多模态感知、机器人操作与具身智能的研究者。
视觉与触觉是机器人感知与操作的两种基础模态，提供互补信息以增强性能。以往方法常采用直接拼接或相加进行模态融合，导致特征整合效果不佳。本文提出ConViTac，一种基于对比表示的视觉-触觉表征学习网络，通过预训练的对比编码器将视觉与触觉输入映射到统一的潜在空间，利用跨模态注意力实现特征耦合，以增强模态对齐。大量实验证明，ConViTac在真实场景下优于当前最优方法，所提CEC机制在材料分类与抓取预测任务中分别提升最高12.0%的准确率。
原文摘要 · Abstract (English)
Vision and touch are two fundamental sensory modalities for robots, offering complementary information that enhances perception and manipulation tasks. Previous research has attempted to jointly learn visual-tactile representations to extract more meaningful information. However, these approaches often rely on direct combination, such as feature addition and concatenation, for modality fusion, which tend to result in poor feature integration. In this paper, we propose ConViTac, a visual-tactile representation learning network designed to enhance the alignment of features during fusion using contrastive representations. Our key contribution is a Contrastive Embedding Conditioning (CEC) mechanism that leverages a contrastive encoder pretrained through self-supervised contrastive learning to project visual and tactile inputs into unified latent embeddings. These embeddings are used to couple visual-tactile feature fusion through cross-modal attention, aiming at aligning the unified representations and enhancing performance on downstream tasks. We conduct extensive experiments to demonstrate the superiority of ConViTac in real world over current state-of-the-art methods and the effectiveness of our proposed CEC mechanism, which improves accuracy by up to 12.0% in material classification and grasping prediction tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。