arXiv:2607.13013cs.AIcs.SD2026-07

用扩散模型并行生成语音转录,8步完成,误差仅6.6%

Audio-Native Speech Recognition with a Frozen Discrete-Diffusion Language Model

论文配图:Audio-Native Speech Recognition with a Frozen Discrete-Diffusion Language Model
图 1 · 摘自论文原文
  • 用离散扩散模型替代自回归解码,全句并行重构
  • 在LibriSpeech上达6.6%词错误率,推理仅需8步
  • 仅训练420万参数,适配多语言语音识别

自动语音识别主要依赖逐词生成的自回归解码器。本文探索使用离散扩散语言模型进行语音转录,通过少量去噪步骤并行优化整段文本。我们为260亿参数的DiffusionGemma(一种基于统一随机采样离散扩散的专家混合模型)构建了音频原生接口,采用冻结的Whisper编码器提取声学特征,轻量级投影器将其映射至模型嵌入空间,并通过低秩适配器使冻结主干网络能关注新模态。仅训练约420万参数(占主干0.16%)。发现自然训练目标因梯度路径受限而无法有效对齐音频;通过在冻结输出头处引入连接时序分类损失,打破此僵局。最终模型在LibriSpeech test-clean上达到6.6%词错误率,无论语句长短,均能在约8步内完成并行转录,且仅用一个在六种语言上训练的适配器,在英语、印地语和中文上均表现良好。

原文摘要 · Abstract (English)

Automatic speech recognition is dominated by autoregressive decoders that emit one token at a time. We ask whether a discrete diffusion language model can transcribe speech instead, refining a whole transcript in parallel over a small number of denoising steps. We train an audio-native interface for DiffusionGemma, a 26B mixture-of-experts model that generates text by uniform, random-token discrete diffusion rather than the absorbing-mask scheme common to recent diffusion language models. A frozen Whisper encoder supplies acoustic features, a lightweight projector maps them into the model embedding space, and low-rank adapters let the frozen backbone attend to the new modality. About 42M parameters are trained, which is 0.16 percent of the backbone. We find that the natural training objectives fail to ground the audio because their gradient reaches the projector only through attention that has already dismissed it. A connectionist temporal classification loss applied through the frozen output head breaks this deadlock. The resulting model reaches 6.6 percent word error rate on LibriSpeech test-clean, transcribes in roughly eight parallel steps regardless of utterance length, and uses a single adapter trained on six languages, which we evaluate here on English, Hindi, and Mandarin.

语音识别扩散模型多语言并行生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。