arXiv:2505.14270cs.CV2025-05被引 2

用视觉数据增强触觉理解,让模型学会从外观猜材质。

RA-Touch: Retrieval-Augmented Touch Understanding with Enriched Visual Data

  • 用触觉描述重写视觉数据,让图像带触感语义。
  • 检索相关视觉文本提升触觉感知准确率,优于已有方法。
  • 适合做多模态感知、触觉理解的开发者参考。

视觉-触觉感知旨在理解物体的触觉属性,如纹理、柔软度和刚性。但该领域进展缓慢,因采集触觉数据成本高且耗时。我们观察到,外观不同的物体可能具有相似的表面纹理或材料特性,例如皮革沙发与皮革外套外观不同却触感相似。这表明仅通过视觉中的材料线索即可指导触觉理解,无需直接触觉标注。本文提出RA-Touch,一种基于检索增强的框架,通过引入富含触觉语义的视觉数据来提升视觉-触觉感知能力。我们对大规模视觉数据集进行精细重标注,加入聚焦触觉的描述,使模型可访问传统视觉数据中缺失的触觉语义。核心挑战在于如何有效利用这些带触觉意识的外部描述。RA-Touch通过检索与触觉输入对齐的视觉-文本表示,并融合相关信息,聚焦于纹理与材料特性。在TVL基准测试中,本方法超越现有方法,证明了基于检索的视觉复用在触觉理解中的潜力。代码已公开于https://aim-skku.github.io/RA-Touch。

原文摘要 · Abstract (English)

Visuo-tactile perception aims to understand an object's tactile properties, such as texture, softness, and rigidity. However, the field remains underexplored because collecting tactile data is costly and labor-intensive. We observe that visually distinct objects can exhibit similar surface textures or material properties. For example, a leather sofa and a leather jacket have different appearances but share similar tactile properties. This implies that tactile understanding can be guided by material cues in visual data, even without direct tactile supervision. In this paper, we introduce RA-Touch, a retrieval-augmented framework that improves visuo-tactile perception by leveraging visual data enriched with tactile semantics. We carefully recaption a large-scale visual dataset with tactile-focused descriptions, enabling the model to access tactile semantics typically absent from conventional visual datasets. A key challenge remains in effectively utilizing these tactile-aware external descriptions. RA-Touch addresses this by retrieving visual-textual representations aligned with tactile inputs and integrating them to focus on relevant textural and material properties. By outperforming prior methods on the TVL benchmark, our method demonstrates the potential of retrieval-based visual reuse for tactile understanding. Code is available at https://aim-skku.github.io/RA-Touch

触觉理解多模态检索增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。