arXiv:2605.23954cs.CLcs.AI2026-05

用干净音频引导噪声下的语音大模型,提升抗噪可靠性。

EchoDistill:Alignment Noisy-to-Clean Self-Distillation for Robust Audio LLMs

论文配图:EchoDistill:Alignment Noisy-to-Clean Self-Distillation for Robust Audio LLMs
图 1 · 摘自论文原文
  • 通过冻结的纯净音频教师引导噪声学生,对齐语义轨迹。
  • 在强噪声下平均提升4.18%的语义恢复率(GSR)。
  • 无需额外计算开销,适合部署于真实嘈杂场景的语音模型。

语音大语言模型(ALLMs)极易受现实世界噪声影响,导致严重语义偏移和幻觉。现有方法多依赖波形级声学增强、答案级监督或内部噪声表征抑制。为此,我们提出 echodistill,一种基于对齐的噪声到纯净自蒸馏框架。该框架利用一个冻结的纯净音频教师,在推理时为噪声输入的学生模型提供语义参考。学生在噪声条件下采样候选响应以暴露其测试行为,再通过组相对策略优化(GRPO)进行优化,其中与教师的词元级一致性作为奖励增益。通过将学生候选响应与纯净语义证据对齐,并施加音频感知奖励塑造,该方法促使推理轨迹既准确又真正具备声学基础。echodistill 在不引入额外推理成本的前提下,显著提升了语音大模型在复杂噪声下的语义可靠性和任务表现。大量实验表明:(I) 与最强基线相比,echodistill 在强噪声下平均提升4.18%的语义恢复率(GSR)。(II) 在 Qwen-Omni 上的消融实验显示,相比仅使用GRPO的变体,echodistill 平均提升3.02%的准确率(Acc)、3.89%的噪声场景性能(Noisy)和4.53%的语义恢复率(GSR)。代码已公开于 https://anonymous.4open.science/r/echodistill-10DE。

原文摘要 · Abstract (English)

Audio Large Language Models (ALLMs) are highly vulnerable to real-world noise, which often induces severe semantic drift and hallucinations. Existing robustness methods primarily rely on waveform-level acoustic enhancement, answer-level supervision, or the internal suppression of noise representations. To address these issues, we propose echodistill, an alignment-based noisy-to-clean self-distillation framework. Echodistill leverages a frozen clean-audio teacher to provide semantic references for an inference-time noisy-audio student. Specifically, the student samples candidate responses under noisy conditions to expose its test-time behavior. These trajectories are then optimized via group-relative policy optimization (GRPO), where the token-level consistency with the teacher acts as a reward bonus. By aligning the noisy student's candidate responses with clean semantic evidence, and applying audio-aware reward shaping, our method encourages reasoning trajectories that are both correct and genuinely acoustically grounded. Echodistill significantly improves the semantic reliability and task performance of Audio LLMs under complex noise, without introducing any additional inference costs. Extensive experiments show that: (I) Compared with the strongest baseline, echodistill achieves average improvements of 4.18\%$\uparrow$ in GSR under strong noise. (II) Ablation results on Qwen-Omni further show that echodistill improves over the GRPO-only variant by 3.02\%$\uparrow$ in Acc, 3.89\%$\uparrow$ in Noisy, and 4.53\%$\uparrow$ in GSR on average. Our codes are available at https://anonymous.4open.science/r/echodistill-10DE.

语音大模型抗噪训练自蒸馏语义对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。