用视觉信息提升双语语音模型性能，缩小多语言差距。
Leveraging Audio-Visual Data to Reduce the Multilingual Gap in Self-Supervised Speech Models
- 在双语语音模型中引入有限视觉线索进行跨模态对齐。
- 零样本语音辨识任务上，多语言性能差距从31.5%降至8.04%。
- 适合研究多语言语音表征与跨模态学习的学者参考。
自监督学习（SSL）在语音表示学习方面取得了显著进展，如wav2vec 2.0和HuBERT在单语言场景下已达到顶尖水平。然而，在多语言设置中，尤其是仅含两种语言的双语场景下，多语言SSL模型在每种语言上的表现仍逊于单语言模型。本文提出一种新方法：在双语语音SSL模型中引入有限的视觉信息作为辅助。实验表明，视觉对齐对单语言和双语模型均有提升，尤其在双语模型上效果显著，将零样本语音辨识任务中的多语言性能差距从音频仅模型的31.5%降低至8.04%。
原文摘要 · Abstract (English)
Self-supervised learning (SSL) has made significant advances in speech representation learning. Models like wav2vec 2.0 and HuBERT have achieved state-of-the-art results in tasks such as speech recognition, particularly in monolingual settings. However, multilingual SSL models tend to underperform their monolingual counterparts on each individual language, especially in multilingual scenarios with few languages such as the bilingual setting. In this work, we investigate a novel approach to reduce this performance gap by introducing limited visual grounding into bilingual speech SSL models. Our results show that visual grounding benefits both monolingual and bilingual models, with especially pronounced gains for the latter, reducing the multilingual performance gap on zero-shot phonetic discrimination from 31.5% for audio-only models to 8.04% with grounding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。