让小模型学会听懂语音，减少语言偏见
CAAD: Contrastive Audio-Aware Distillation for Efficient Speech Language Models
- 用对比学习将大模型的听觉推理能力蒸馏到小模型
- 在Dynamic-SUPERB上提升8%性能，降低语言偏见
- 同步教师强迫策略加速训练，适合语音理解任务
语音语言模型具备推理能力，但常因参数量大且过度依赖语言先验而忽视语音特征。虽然对比解码通过对比音频感知与纯文本得分可增强语义对齐，但会增加推理延迟。本文提出对比音频感知蒸馏（CAAD），将教师模型的对比推理机制内化到学生模型权重中。为克服双路径逐标记对比蒸馏带来的高计算开销，引入同步教师强迫策略，基于统一伪真实标签，实现教师模型对比分布的全序列同步生成，使学生模型高效蒸馏音频感知信号。整体上，CAAD在Dynamic-SUPERB上相较标准知识蒸馏获得约8%相对提升，并成功降低MCR-BENCH中的语言偏见。
原文摘要 · Abstract (English)
Speech Language Models achieve reasoning capabilities, but are often hindered by massive parameter counts and a tendency to prioritize linguistic priors over acoustic features. While contrastive decoding enhances grounding by contrasting audio-aware and text-only logits, it increases inference latency. We propose Contrastive Audio-Aware Distillation (CAAD), a framework that internalizes the teacher's contrastive reasoning into the student model's weights. To overcome the high computational training overhead in the dual-path token-by-token contrastive distillation process, we introduce a synchronized teacher-forcing strategy. Anchored by unified Pseudo-Ground Truths, this mechanism enables simultaneous full-sequence generation of the teacher's contrastive distributions, allowing student to distill the audio-aware signal efficiently. Overall, CAAD yields a ~8% relative gain over standard knowledge distillation on Dynamic-SUPERB and successfully reduces linguistic bias in MCR-BENCH.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。