用连续向量生成语音识别与翻译,提升模型对语义的统一理解。
Speech Meets ELF: Audio Conditional Continuous-Target Diffusion for Speech Recognition and Translation

- 基于预训练语言流模型,将语音编码后连续映射到文本空间。
- 在LibriSpeech和CoVoST2上达到与离散生成相当的识别与翻译性能。
- 揭示语音识别与翻译错误本质相同,源于连续空间中的语义混淆。
语音转文本(S2T)系统通常生成离散文本标记,而连续目标语言建模在连续空间中进行生成,其在S2T中的潜力尚未探索。为此,我们提出ELF-S2T,一种音频条件化的连续目标生成模型。基于预训练的嵌入语言流(Embedded Language Flows, ELF)主干,ELF-S2T通过冻结的Whisper编码器和单层线性投影器处理语音,将音频条件拼接至噪声文本潜空间,实现上下文感知的流匹配去噪。为防止模型过度依赖预训练文本上下文,训练时引入音频强制机制,并在推理时通过无分类器引导增强音频条件。在LibriSpeech和CoVoST2上的实验表明,ELF-S2T实现了具有竞争力的自动语音识别(ASR)和语音翻译(S2TT)性能。关键的是,误差分析显示,尽管ASR与S2TT错误表面差异明显,但均源于连续潜空间中的近邻混淆。这一发现与连续表示生成范式自然契合,表明识别与翻译共享相同的语义映射过程。代码与预训练模型已公开于https://github.com/Sslnon/ELF-S2T。
原文摘要 · Abstract (English)
Speech-to-text (S2T) systems for recognition (ASR) and translation (S2TT) typically generate discrete text tokens. In contrast, continuous-target language modelling performs generation in a continuous space, yet its potential for S2T remains unexplored. To bridge this gap, we propose ELF-S2T, an audio-conditioned continuous-target generative model for S2T. Built upon the pre-trained Embedded Language Flows (ELF) backbone, ELF-S2T processes speech via a frozen Whisper encoder and a single linear projector, prepending the resulting audio condition to the noisy text latent for in-context, flow-matching denoising. To prevent the model from over-relying on its pre-trained text context, we introduce audio forcing during training, and further amplify the audio condition via classifier-free guidance at inference. Experiments on LibriSpeech and CoVoST2 show that ELF-S2T achieves competitive ASR and S2TT performance. Crucially, our error analysis reveals that, although ASR and S2TT errors look very different on the surface, both stem from the same underlying cause, a close distance confusion in the continuous latent space. This finding naturally aligns with the continuous representation generation paradigm, indicating a common semantic mapping process beneath recognition and translation. Our code and pretrained models are publicly available at https://github.com/Sslnon/ELF-S2T.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。