用扩散模型提速语音识别,速度提升4.44倍且准确率不降。
dLLM-ASR: A Faster Diffusion LLM-based Framework for Speech Recognition
- 将扩散模型改造成语音识别专用,用声学先验引导去噪过程。
- 通过动态剪枝和置信度提前退出,实现按需计算,加速推理。
- 适合追求低延迟语音识别的工业场景,尤其对长语音优化明显。
基于大语言模型(LLM)的自动语音识别(ASR)系统利用预训练解码器获得优异性能,但其逐标记生成机制导致推理延迟随序列长度线性增长。离散扩散大语言模型(dLLM)可实现高质量并行生成,但直接用于ASR存在根本性不匹配:开放文本生成与声学条件化转录之间存在差异,导致从纯噪声去噪、生成长度固定、去噪步数不可调等冗余问题。本文提出dLLM-ASR,将dLLM解码重构为先验引导的自适应去噪过程,利用ASR先验初始化去噪并锚定序列长度。在此基础上,长度自适应剪枝动态移除冗余标记,置信度驱动去噪使已收敛标记提前退出,实现标记级自适应计算。实验表明,dLLM-ASR在识别准确率上媲美自回归式LLM-ASR,推理速度提升4.44倍,建立了一种高效实用的语音识别新范式。
原文摘要 · Abstract (English)
Automatic speech recognition (ASR) systems based on large language models (LLMs) achieve superior performance by leveraging pretrained LLMs as decoders, but their token-by-token generation mechanism leads to inference latency that grows linearly with sequence length. Meanwhile, discrete diffusion large language models (dLLMs) offer a promising alternative, enabling high-quality parallel sequence generation with pretrained decoders. However, directly applying native text-oriented dLLMs to ASR leads to a fundamental mismatch between open-ended text generation and the acoustically conditioned transcription paradigm required by ASR. As a result, it introduces unnecessary difficulty and computational redundancy, such as denoising from pure noise, inflexible generation lengths, and fixed denoising steps. We propose dLLM-ASR, an efficient dLLM-based ASR framework that formulates dLLM's decoding as a prior-guided and adaptive denoising process. It leverages an ASR prior to initialize the denoising process and provide an anchor for sequence length. Building upon this prior, length-adaptive pruning dynamically removes redundant tokens, while confidence-based denoising allows converged tokens to exit the denoising loop early, enabling token-level adaptive computation. Experiments demonstrate that dLLM-ASR achieves recognition accuracy comparable to autoregressive LLM-based ASR systems and delivers a 4.44$\times$ inference speedup, establishing a practical and efficient paradigm for ASR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。