分析多模态大模型如何统一文本与语音的跨语言表示
How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representations
- 通过对比不同语言的文本和语音输入，研究模型内部表征演化
- 语音与文本的跨模态差异随层数加深而缩小，但初期层有专属性能
- 当前方法对低资源语言效果有限，语音比文本跨语言差异更大
多模态基础模型旨在构建一个抽象掉语言语法或模态差异的统一表征空间。我们研究了三个近期模型的内部表征，分析了跨语言、跨模态语义等价句子的模型激活。结果表明：1）跨模态表征在模型深层趋于收敛，初始层则分别专注文本与语音处理；2）长度适配对缩小文本与语音间的跨模态差距至关重要，但现有方法在低资源语言上效果有限；3）语音的跨语言差异大于文本；4）对于未显式训练为模态无关的模型，模态差异大于语言差异。
原文摘要 · Abstract (English)
Multimodal foundation models aim to create a unified representation space that abstracts away from surface features like language syntax or modality differences. To investigate this, we study the internal representations of three recent models, analyzing the model activations from semantically equivalent sentences across languages in the text and speech modalities. Our findings reveal that: 1) Cross-modal representations converge over model layers, except in the initial layers specialized at text and speech processing. 2) Length adaptation is crucial for reducing the cross-modal gap between text and speech, although current approaches' effectiveness is primarily limited to high-resource languages. 3) Speech exhibits larger cross-lingual differences than text. 4) For models not explicitly trained for modality-agnostic representations, the modality gap is more prominent than the language gap.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。