用自剪枝门控实现语音大模型单阶段高效压缩，参数减少超60%且精度不变
Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gates
- 通过层级共享的自剪枝门控实现神经元级稀疏化，与模型训练同步进行
- wav2vec2.0-base和HuBERT-large参数量分别减少65%和60%，测试集错误率无显著上升
- 压缩速度比现有方法快至少25%，在4.26倍压缩比下达到7.05%最低错误率
本文提出一种新型语音基础模型压缩方法，将模型剪枝与参数更新整合于单一阶段。采用每层仅含一个可学习阈值的层级共享自剪枝门控，与未压缩模型联合训练，实现细粒度神经元级剪枝。在LibriSpeech-100hr语料上的实验表明，该方法使wav2vec2.0-base和HuBERT-large模型参数量分别减少65%和60%，在test-clean数据集上未引入统计显著的词错误率（WER）增长。相比已有方法，在相近压缩比4.26x下，本方法达到7.05%最低WER，且压缩时间至少减少25%。
原文摘要 · Abstract (English)
This paper presents a novel approach for speech foundation models compression that tightly integrates model pruning and parameter update into a single stage. Highly compact layer-level tied self-pinching gates each containing only a single learnable threshold are jointly trained with uncompressed models and used in fine-grained neuron level pruning. Experiments conducted on the LibriSpeech-100hr corpus suggest that our approach reduces the number of parameters of wav2vec2.0-base and HuBERT-large models by 65% and 60% respectively, while incurring no statistically significant word error rate (WER) increase on the test-clean dataset. Compared to previously published methods on the same task, our approach not only achieves the lowest WER of 7.05% on the test-clean dataset under a comparable model compression ratio of 4.26x, but also operates with at least 25% less model compression time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。