用扩散语言模型实现灵活顺序语音识别，提升视觉语音识别准确率
Diffusion Large Language Models for Visual Speech Recognition
- 将视觉语音识别转为迭代去噪过程，支持非顺序解码
- 在LRS3上达到19.4%词错误率，性能领先现有方法
- 利用视频时长生成合理长度假设，优化解码结果
现有视觉语音识别（VSR）系统多依赖从左到右的自回归解码，会在上下文不足时过早做出判断。本文提出首个基于扩散大语言模型（DLLM）的VSR框架DLLM-VSR，将文本生成建模为迭代掩码去噪，支持灵活顺序解码。通过置信度驱动的去掩码策略，高置信度位置优先确定，并作为双向上下文用于修正模糊部分。为适配DLLM于VSR，设计两阶段掩码去噪训练：先对齐视觉与文本内容，再建模序列长度。实验发现，当推理时提供真实语句长度（上界设定）可显著提升性能，表明当前模型在长度预测上仍有差距。为此，提出长度引导候选解码，利用视频时长构建合理的语句长度假设，并基于长度合理性与解码置信度重排序候选结果。该方法在LRS3数据集上实现19.4%的词错误率，是仅使用LRS3进行标注训练的方法中的最先进水平。
原文摘要 · Abstract (English)
Existing Visual Speech Recognition (VSR) systems commonly rely on left-to-right autoregressive decoding, which can force premature decisions on visually ambiguous tokens before sufficient context is available. We propose DLLM-VSR, to the best of our knowledge, the first Diffusion Large Language Model (DLLM)-based VSR framework, formulating transcription as iterative masked denoising with flexible-order decoding. With confidence-based unmasking, DLLM-VSR commits high-confidence positions early and uses the committed tokens as bidirectional context to refine ambiguous ones. To adapt DLLMs to VSR, we introduce a two-stage masked-denoising training strategy that separates visual-to-text content alignment from length modeling. We further observe a performance gap compared with an upper-bound setting where the ground-truth transcript length is provided at inference, allowing the model to focus on transcript content decoding. To reduce this gap, we develop length-guided candidate decoding, which uses video duration to construct plausible transcript-length hypotheses and reranks the decoded candidates using length plausibility and decoding confidence. The proposed method achieves a 19.4% word error rate on LRS3, establishing state-of-the-art performance among methods using only LRS3 as labeled training data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。