将触觉振动信号转化为自然语言描述，首次实现触觉内容自动解说。
The Language of Touch: Translating Vibrations into Text with Dual-Branch Learning
- 分周期与非周期双分支处理触觉信号结构，动态融合特征提升理解力。
- 在自建数据集上达到领先性能，关键词匹配度和语义一致性显著提升。
- 适合触觉交互、虚拟现实及具身智能领域的研究人员参考。
IEEE P1918.1工作组对触觉数据的标准化推动了其在虚拟现实、人机交互和具身人工智能中的应用。然而，触觉信号的语义理解仍是未解难题。本文首次提出触觉字幕生成任务，即从触觉信号生成自然语言描述。我们提出ViPAC方法，针对触觉信号的混合周期-非周期特性及缺乏空间语义的问题，采用双分支结构分离周期与非周期成分，并通过动态融合机制自适应整合特征。同时引入正交性约束与加权正则化，确保特征互补与融合一致性。此外，我们构建了首个触觉-文本配对数据集LMT108-CAP，基于LMT-108数据集，利用GPT-4o为每张表面图像生成5个受约束的描述。实验表明，ViPAC显著优于从音频和图像字幕迁移而来的基线方法，在词汇准确性和语义对齐方面表现更优。
原文摘要 · Abstract (English)
The standardization of vibrotactile data by IEEE P1918.1 workgroup has greatly advanced its applications in virtual reality, human-computer interaction and embodied artificial intelligence. Despite these efforts, the semantic interpretation and understanding of vibrotactile signals remain an unresolved challenge. In this paper, we make the first attempt to address vibrotactile captioning, {\it i.e.}, generating natural language descriptions from vibrotactile signals. We propose Vibrotactile Periodic-Aperiodic Captioning (ViPAC), a method designed to handle the intrinsic properties of vibrotactile data, including hybrid periodic-aperiodic structures and the lack of spatial semantics. Specifically, ViPAC employs a dual-branch strategy to disentangle periodic and aperiodic components, combined with a dynamic fusion mechanism that adaptively integrates signal features. It also introduces an orthogonality constraint and weighting regularization to ensure feature complementarity and fusion consistency. Additionally, we construct LMT108-CAP, the first vibrotactile-text paired dataset, using GPT-4o to generate five constrained captions per surface image from the popular LMT-108 dataset. Experiments show that ViPAC significantly outperforms the baseline methods adapted from audio and image captioning, achieving superior lexical fidelity and semantic alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。