对比了语音转文本中两种主流方法的性能，发现交叉注意力不逊于主流的特征拼接法。
Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison
- 比较了特征拼接与交叉注意力两种语音融合方式的性能差异
- 在多语言数据集上，两种方法效果接近，无明显优劣
- 适用于语音识别与语音翻译任务的研究者参考
随着大语言模型在自然语言处理任务中的成功，人们越来越关注将其能力拓展至语音——最普遍的交流形式。目前将语音融入大语言模型的主流方法是密集特征拼接（DFP），即把投影后的语音表示拼接到文本表示前，实现端到端训练并使用语音编码器。这引发了对复杂语音编码器是否必要以及其性能是否优于标准编码器-解码器架构（即交叉注意力）的疑问。本文在多种配置下对比了DFP与交叉注意力，包括CTC压缩、序列级知识蒸馏，覆盖单语、双语和多语言模型。为确保架构对比的公平性，所有模型均从零训练，数据和参数设置保持一致，并在MuST-C v1.0和CoVoST2数据集上测试语音识别（ASR）与语音翻译（ST）任务。尽管DFP被广泛采用，但实验结果并未显示出其相对于交叉注意力的明确优势。
原文摘要 · Abstract (English)
Following the remarkable success of Large Language Models (LLMs) in NLP tasks, there is increasing interest in extending their capabilities to speech -- the most common form of communication. The most widespread approach to integrating speech into LLMs is dense feature prepending (DFP), which prepends the projected speech representations to the textual representations, allowing end-to-end training with a speech encoder. This raises questions about the need for a sophisticated speech encoder for DFP and how its performance compares with a standard encoder-decoder (i.e., cross-attention) architecture. We compare DFP and cross-attention under a variety of configurations, such as CTC compression, sequence-level knowledge distillation, on monolingual, bilingual, and multilingual models. To perform a controlled architectural comparison, we train all models from scratch rather than using large pretrained models and use comparable data and parameter settings, testing speech-to-text recognition (ASR) and translation (ST) on MuST-C v1.0 and CoVoST2 datasets. Despite the wide adoption of DFP, our results do not indicate a clear advantage of DFP over cross-attention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。