用双路语音视觉纠错，让大模型更准识别有干扰的语音
Two Heads Are Better Than One: Audio-Visual Speech Error Correction with Dual Hypotheses
- 让大模型分别处理语音和视觉两种输入，生成独立候选词
- 在噪声环境下纠错效果提升57.7%，远超单一语音方法
- 适合做语音识别鲁棒性研究或跨模态系统开发
本文提出一种新型生成式错误纠正框架DualHyp，用于音视频语音识别（AVSR）。该框架让大语言模型（LLM）基于独立的自动语音识别（ASR）与视觉语音识别（VSR）模型生成的N-best候选结果，在语言空间中直接推理。为提升效果，我们引入RelPrompt——一种噪声感知引导机制，提供各模态的时间可靠性信息，使模型可动态切换关注语音或视觉线索进行修正。在多种噪声场景下，DualHyp在LRS2基准上相较标准ASR基线最高实现57.7%的错误率降低，显著优于仅使用单流输入的GER方法（仅10%提升）。相关代码与包含ASR/VSR候选结果的数据集已开源：https://github.com/sungnyun/dualhyp。
原文摘要 · Abstract (English)
This paper introduces a new paradigm for generative error correction (GER) framework in audio-visual speech recognition (AVSR) that reasons over modality-specific evidences directly in the language space. Our framework, DualHyp, empowers a large language model (LLM) to compose independent N-best hypotheses from separate automatic speech recognition (ASR) and visual speech recognition (VSR) models. To maximize the effectiveness of DualHyp, we further introduce RelPrompt, a noise-aware guidance mechanism that provides modality-grounded prompts to the LLM. RelPrompt offers the temporal reliability of each modality stream, guiding the model to dynamically switch its focus between ASR and VSR hypotheses for an accurate correction. Under various corruption scenarios, our framework attains up to 57.7% error rate gain on the LRS2 benchmark over standard ASR baseline, contrary to single-stream GER approaches that achieve only 10% gain. To facilitate research within our DualHyp framework, we release the code and the dataset comprising ASR and VSR hypotheses at https://github.com/sungnyun/dualhyp.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。