揭秘LoRA如何让Whisper模型识别语音情绪，揭示其内部工作机制。
Behind the Scenes: Mechanistic Interpretability of LoRA-adapted Whisper for Speech Emotion Recognition
- 通过层贡献分析等工具，研究LoRA在语音情感识别中的作用机制。
- 发现早期层保留通用特征，后期才逐步聚焦任务特异性信息。
- 适合对模型可解释性与高效微调感兴趣的语音处理研究人员。
大型预训练语音模型如Whisper具备强大泛化能力，但资源高效适配面临挑战。低秩适应（LoRA）作为一种流行的参数高效微调方法，其在语音任务中的内在机制仍不清晰。本文首次对Whisper编码器中用于语音情感识别（SER）的LoRA进行了系统性的机制可解释性研究。借助层贡献探测、logit-lens检查以及基于奇异值分解（SVD）和中心核对齐（CKA）的表示相似性分析，我们揭示了两个关键机制：一是延迟专业化过程，即早期层保持通用特征，后期才整合任务特异性信息；二是LoRA矩阵间的前向对齐、后向差异化的动态。这些发现阐明了LoRA如何重塑编码器层级结构，为设计高效且可解释的大规模语音模型适配策略提供了实证依据与深层理解。代码已公开于https://github.com/harryporry77/Behind-the-Scenes。
原文摘要 · Abstract (English)
Large pre-trained speech models such as Whisper offer strong generalization but pose significant challenges for resource-efficient adaptation. Low-Rank Adaptation (LoRA) has become a popular parameter-efficient fine-tuning method, yet its underlying mechanisms in speech tasks remain poorly understood. In this work, we conduct the first systematic mechanistic interpretability study of LoRA within the Whisper encoder for speech emotion recognition (SER). Using a suite of analytical tools, including layer contribution probing, logit-lens inspection, and representational similarity via singular value decomposition (SVD) and centered kernel alignment (CKA), we reveal two key mechanisms: a delayed specialization process that preserves general features in early layers before consolidating task-specific information, and a forward alignment, backward differentiation dynamic between LoRA's matrices. Our findings clarify how LoRA reshapes encoder hierarchies, providing both empirical insights and a deeper mechanistic understanding for designing efficient and interpretable adaptation strategies in large speech models. Our code is available at https://github.com/harryporry77/Behind-the-Scenes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。