用噪声级别嵌入提升波斯语语音识别在嘈杂环境下的鲁棒性。
Incorporating Error Level Noise Embedding for Improving LLM-Assisted Robustness in Persian Speech Recognition
- 引入噪声级别嵌入（ELN）量化多候选假设间的语义与词级差异。
- 在混合噪声测试集上将词错误率从31.10%降至24.84%。
- 适合关注低资源语言、噪声鲁棒语音识别的研究者。
自动语音识别（ASR）系统在噪声环境下性能显著下降，这对波斯语等低资源语言尤为严峻。即使先进模型如Whisper在不同信噪比下也难以保持准确率。本研究提出一种结合多候选假设与噪声感知建模的鲁棒纠错框架。基于修改后的Whisper-large解码器，生成5个候选输出，并引入误差级别噪声（ELN）作为表征，捕捉候选间语义与词级不一致，量化噪声引起的语言失真。ELN为大语言模型提供噪声不确定性直接度量，使其在纠错时评估各假设可靠性。对比三种模型：（1）未微调的LLaMA-2-7B；（2）仅用文本候选微调的版本；（3）融合句子与词级ELN嵌入的噪声条件模型。实验显示，所提细调+ELN模型在混合噪声测试集上将词错误率（WER）从原始Whisper的31.10%降至24.84%，显著优于仅文本微调的30.79%，而原始LLaMA-2-7B反而升至64.58%，表明其无法独立纠正波斯语错误。结果验证了结合多假设与噪声感知嵌入对提升波斯语语音识别鲁棒性的有效性。
原文摘要 · Abstract (English)
Automatic Speech Recognition (ASR) systems suffer significant performance degradation in noisy environments, a challenge that is especially severe for low-resource languages such as Persian. Even state-of-the-art models such as Whisper struggle to maintain accuracy under varying signal-to-noise ratios (SNRs). This study presents a robust noise-sensitive ASR error correction framework that combines multiple hypotheses and noise-aware modeling. Using noisy Persian speech, we generate 5-best hypotheses from a modified Whisper-large decoder. Error Level Noise (ELN) is introduced as a representation that captures semantic- and token-level disagreement across hypotheses, quantifying the linguistic distortions caused by noise. ELN thus provides a direct measure of noise-induced uncertainty, enabling the LLM to reason about the reliability of each hypothesis during correction. Three models are evaluated: (1) a base LLaMA-2-7B model without fine-tuning, (2) a fine-tuned variant trained on text-only hypotheses, and (3) a noise-conditioned model integrating ELN embeddings at both sentence and word levels. Experimental results demonstrate that the ELN-conditioned model achieves substantial reductions in Word Error Rate (WER). Specifically, on the challenging Mixed Noise test set, the proposed Fine-tuned + ELN (Ours) model reduces the WER from a baseline of 31.10\% (Raw Whisper) to 24.84\%, significantly surpassing the Fine-tuned (No ELN) text-only baseline of 30.79\%, whereas the original LLaMA-2-7B model increased the WER to 64.58\%, demonstrating that it is unable to correct Persian errors on its own. This confirms the effectiveness of combining multiple hypotheses with noise-aware embeddings for robust Persian ASR in noisy real-world scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。