提出一种实时诊断方法,揭示语音增强如何损害识别性能。
Where Speech Enhancement Hurts Recognition: An Inference Time Polar Projection Diagnosis
- 通过极坐标投影分离幅度与相位影响,定位干扰识别的关键成分。
- 发现幅度强度是影响识别的核心因素,相位修正无实际帮助。
- 无需重训练即可适配各类语音助手,提升实用价值。
语音增强(SE)可显著提升听觉质量,但增强后的语音未必改善自动语音识别(ASR)性能。现有方法如联合重训或混合输入虽能缓解失配问题,但对损害原因的解释仍停留在定性层面,无法定位具体组件。本文提出推理时极坐标投影法,针对STFT域掩码$M=Ae^{jϕ}$,构造$M_{α,γ}=A^αe^{jγϕ}$,其中$α$控制幅度强度,$γ$控制相位修正。在冻结的SE与ASR模型上扫描参数,将识别退化转化为可测量的幅度与相位效应。分析表明,幅度强度是决定性因素,而相位修正未带来识别增益。最优幅度强度依赖于识别器:波形输入的wav2vec2.0偏好强校正,而日志梅尔输入、抗噪性强的Whisper则偏好弱校正。该投影方法可直接用于任何基于STFT掩码的增强前端,无需重训增强器或识别器,适用于依赖增强语音的语音助手与智能代理。
原文摘要 · Abstract (English)
Speech enhancement (SE) can substantially improve perceptual quality, yet enhanced speech does not necessarily improve automatic speech recognition (ASR). Existing remedies, such as retraining the enhancer jointly with recognizer or interpolating enhanced speech with the noisy input, can mitigate this mismatch, but common explanations such as artifacts and over-suppression remain qualitative and do not localize which enhancement component harms recognition. We propose inference time polar projection, a diagnosis for STFT domain enhancement. Given a mask $M=Ae^{jϕ}$, polar projection forms $M_{α,γ}=A^αe^{jγϕ}$, where $α$ controls magnitude strength and $γ$ controls phase correction. Sweeping these controls on frozen SE and ASR models turns ASR degradation into measurable magnitude and phase effects. Our projection analysis shows that magnitude strength is the operative axis, while estimated phase correction provides no recognition benefit. The optimal magnitude strength is recognizer dependent: waveform-input wav2vec2.0 favors strong correction, whereas log-Mel-input, noise-robust Whisper prefers weaker correction. Finally, the projection provides a simple mitigation for any SE front end in the STFT mask domain, without retraining either the enhancer or the recognizer, making it directly useful for voice assistants and agents that rely on enhanced speech.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。