剪掉语音模型两层,用低秩微调补救,参数减少14%仍保持高精度。
On the Role of Encoder Depth: Pruning Whisper and LoRA Fine-Tuning in SLAM-ASR
- 剪掉Whisper编码器两层,保留核心声学特征。
- 剪枝+LoRA微调使错误率仅增2-4%,参数量降7-14%。
- 对英语荷兰语效果好,丹麦语补偿弱且插入错误增多。
近年来,自动语音识别(ASR)因大规模预训练模型和端到端架构(如SLAM-ASR)而快速发展。其中,Whisper语音编码器作为声学骨干,提供鲁棒的声学表征。尽管已有研究探索完整Whisper编码器-解码器结构的模型剪枝,但在SLAM-ASR设置下的影响仍不明确。本文分析了在SLAM-ASR中对Whisper编码器进行层剪枝的影响,并进一步考察了基于LoRA的微调在恢复剪枝导致性能下降方面的有效性。实验涵盖三种Whisper变体(Small、Medium、Large-v2)、三种语言(丹麦语、荷兰语、英语,代表不同资源水平),以及超过200次训练运行。结果表明:剪掉两个编码器层仅导致2-4%的词错误率(WER)增加;结合剪枝与LoRA适配,性能持续优于未剪枝基线,同时总参数量减少7-14%。误差分析显示,LoRA主要通过语言模型的语义先验进行补偿,使荷兰语和英语的总词错误减少11-21%,替换与删除错误降幅最大;但对低资源丹麦语,补偿效果较弱(减少4-7%),且引入更多插入错误,表明补偿效果依赖于大语言模型(LLM)的预训练语言能力及可用训练数据。
原文摘要 · Abstract (English)
Automatic speech recognition (ASR) has advanced rapidly in recent years, driven by large-scale pretrained models and end-to-end architectures such as SLAM-ASR. A key component of SLAM-ASR systems is the Whisper speech encoder, which provides robust acoustic representations. While model pruning has been explored for the full Whisper encoder-decoder architecture, its impact within the SLAM-ASR setting remains under-investigated. In this work, we analyze the effects of layer pruning in the Whisper encoder when used as the acoustic backbone of SLAM-ASR. We further examine the extent to which LoRA-based fine-tuning can recover performance degradation caused by pruning. Experiments conducted across three Whisper variants (Small, Medium, Large-v2), three languages representing distinct resource levels (Danish, Dutch, English), and over 200 training runs demonstrate that pruning two encoder layers causes only 2-4% WER degradation, and that combining this pruning with LoRA adaptation consistently outperforms the unpruned baseline while reducing total parameters by 7-14%. Moreover, our error analysis reveals that LoRA primarily compensates through the language model's linguistic priors, reducing total word errors by 11-21% for Dutch and English, with substitutions and deletions showing the largest reductions. However, for low-resource Danish, the reduction is smaller (4-7%), and LoRA introduces increased insertion errors, indicating that compensation effectiveness depends on the LLM's pre-existing language proficiency and available training data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。