用注意力加权核对齐，让小模型学懂大模型的跨模态情感识别能力。
Attention-weighted Centered Kernel Alignment for Knowledge Distillation in Large Audio-Language Models Applied to Speech Emotion Recognition
- 通过注意力加权核对齐，解决音频嵌入维度不匹配问题。
- 在三个数据集上将84亿参数模型压缩为11亿参数，性能反超教师模型。
- 适合资源受限场景下的语音情感识别模型部署。
大型音频-语言模型（LALMs）推动了语音情感识别（SER）的发展，但其庞大体积限制了在资源受限环境中的部署。知识蒸馏虽能有效压缩模型，但现有方法对跨模态投影模块（Projector）的蒸馏仍研究不足，且因特征维度差异导致对齐困难。本文提出PL-Distill框架，结合投影层蒸馏（PDist）对齐音频嵌入，以及输出层蒸馏（LDist）对齐音频与文本模态的输出概率。PDist引入注意力加权中心核对齐（Attention-weighted Centered Kernel Alignment），突出重要时间步并缓解维度不匹配。LDist则最小化教师与学生模型输出逻辑斯蒂之间的KL散度。在IEMOCAP、RAVDESS和SAVEE数据集上，PL-Distill将8.4B参数教师模型压缩至1.1B参数学生模型，所有指标均持续优于教师模型、现有预训练模型及主流蒸馏基线。
原文摘要 · Abstract (English)
The emergence of Large Audio-Language Models (LALMs) has advanced Speech Emotion Recognition (SER), but their size limits deployment in resource-constrained environments. While Knowledge Distillation is effective for LALM compression, existing methods remain underexplored in distilling the cross-modal projection module (Projector), and often struggle with alignment due to differences in feature dimensions. We propose PL-Distill, a KD framework that combines Projector-Level Distillation (PDist) to align audio embeddings and Logits-Level Distillation (LDist) to align output logits. PDist introduces Attention-weighted Centered Kernel Alignment, a novel approach we propose to highlight important time steps and address dimension mismatches. Meanwhile, LDist minimizes the Kullback-Leibler divergence between teacher and student logits from audio and text modalities. On IEMOCAP, RAVDESS, and SAVEE, PL-Distill compresses an 8.4B-parameter teacher to a compact 1.1B-parameter student, consistently outperforming the teacher, state-of-the-art pretrained models, and other KD baselines across all metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。