arXiv:2503.06860cs.CV2025-03被引 2

提出无泄露评估与无参考指标,提升触觉图像生成的泛化能力

Towards Generalization of Tactile Image Generation: Reference-Free Evaluation in a Leakage-Free Setting

  • 用文本作为中间模态,结合材料描述增强触觉特征捕捉
  • 在两个数据集上实现更优性能与更强泛化性,避免训练测试样本重叠带来的虚假提升
  • 设计全新无参考指标,可准确评估触觉生成质量,适合触觉建模与机器人研究者

触觉感知依赖直接物理接触,在计算机视觉、机器人和多模态学习中至关重要。由于真实触觉数据稀缺且采集成本高,生成合成触觉图像可有效扩展实测数据。然而,如何在生成中保持鲁棒泛化性——尤其是捕捉细微的、材料特异的接触特征——仍具挑战。本文指出,当前常用数据集中训练与测试样本存在重叠,导致性能指标虚高,掩盖了模型真实泛化能力。为此,我们提出无泄露评估协议,并设计四种专用于触觉生成的无参考度量:TMMD、I-TMMD、CI-TMMD 和 D-TMMD。同时,提出一种视觉到触觉生成方法,通过在训练中引入简洁的材料特定描述作为文本中间模态,以更好捕获关键触觉特征。在两个主流跨模态触觉数据集 Touch and Go 与 HCT 上的实验表明,该方法在无泄露设置下实现了更优性能与更强泛化性。

原文摘要 · Abstract (English)

Tactile sensing, which relies on direct physical contact, is critical for human perception and underpins applications in computer vision, robotics, and multimodal learning. Because tactile data is often scarce and costly to acquire, generating synthetic tactile images provides a scalable solution to augment real-world measurements. However, ensuring robust generalization in synthesizing tactile images-capturing subtle, material-specific contact features-remains challenging. We demonstrate that overlapping training and test samples in commonly used datasets inflate performance metrics, obscuring the true generalizability of tactile models. To address this, we propose a leakage-free evaluation protocol coupled with novel, reference-free metrics-TMMD, I-TMMD, CI-TMMD, and D-TMMD-tailored for tactile generation. Moreover, we propose a vision-to-touch generation method that leverages text as an intermediate modality by incorporating concise, material-specific descriptions during training to better capture essential tactile features. Experiments on two popular visuo-tactile datasets, Touch and Go and HCT, show that our approach achieves superior performance and enhanced generalization in a leakage-free setting.

触觉生成多模态无参考评估泛化性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。