解决语音大模型在真实对话中因历史信息不准导致的识别偏差问题
From Oracle to Noisy Context: Mitigating Contextual Exposure Bias in Speech-LLMs
- 用Whisper生成的假历史替代理想历史进行训练，模拟真实场景
- 引入上下文丢弃和偏好优化，使模型更抗错误历史干扰
- 在真实对话场景下显著降低错误率，适合部署于实际语音助手
使用语音大模型进行上下文自动语音识别时，通常在训练中采用理想的历史对话内容，但在推理时却依赖有误的历史信息，造成上下文通道的训练-测试不匹配，称为上下文暴露偏差。为此，本文提出统一训练框架：(i) 教师错误知识——以Whisper large-v3的预测结果作为训练时的历史；(ii) 上下文丢弃——防止模型过度依赖历史；(iii) 在筛选出的失败案例上使用直接偏好优化（DPO）。在TED-LIUM 3（域内）和零样本LibriSpeech（域外）数据集上的实验表明，在预测历史解码下性能持续提升。当使用两句话历史时，使用Whisper假设进行微调可将词错误率从5.59%（理想历史训练）降至5.47%，DPO进一步优化至5.17%。在无关上下文攻击下，DPO表现最优，错误率仅从5.17%升至5.63%，显示更强鲁棒性。代码与模型已开源。
原文摘要 · Abstract (English)
Contextual automatic speech recognition (ASR) with Speech-LLMs is typically trained with oracle conversation history, but relies on error-prone history at inference, causing a train-test mismatch in the context channel that we term contextual exposure bias. We propose a unified training framework to improve robustness under realistic histories: (i) Teacher Error Knowledge by using Whisper large-v3 hypotheses as training-time history, (ii) Context Dropout to regularize over-reliance on history, and (iii) Direct Preference Optimization (DPO) on curated failure cases. Experiments on TED-LIUM 3 (in-domain) and zero-shot LibriSpeech (out-of-domain) show consistent gains under predicted-history decoding. With a two-utterance history as context, SFT with Whisper hypotheses reduce WER from 5.59% (oracle-history training) to 5.47%, and DPO further improves to 5.17%. Under irrelevant-context attacks, DPO yields the smallest degradation (5.17% -> 5.63%), indicating improved robustness to misleading context. Our code and models are published on https://github.com/XYGuo1996/Contextual_Speech_LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。