通过结构化稀疏与自适应剪枝，让Whisper模型更轻量高效。
Structured Sparsity and Weight-adaptive Pruning for Memory and Compute efficient Whisper models
- 用稀疏分组LASSO正则化实现结构化稀疏，减少计算量。
- 在不损失识别精度前提下，参数减少超30%，内存和算力降低15%以上。
- 适合边缘设备部署，尤其对资源受限场景的语音识别有实用价值。
Whisper模型在语音识别上取得显著进展，但其庞大体积限制了在资源受限边缘设备上的部署。本文提出一种框架，通过稀疏分组LASSO正则化作为损失约束，引入结构化稀疏以降低浮点运算次数（FLOPs）。进一步设计基于权重统计的自适应剪枝算法，并构建专用文本归一化器用于词错误率（WER）评估。在Common Voice 11.0 Hindi数据集上，Whisper-small实现参数减少35.4%、内存消耗降低14.25%、FLOPs减少18.5%；Whisper-medium实现参数减少31%、内存降低15.29%、FLOPs减少16.95%；且相比当前主流的迭代幅度剪枝方法，多剪枝18.7%参数，同时降低12.31的WER。
原文摘要 · Abstract (English)
Whisper models have achieved remarkable progress in speech recognition; yet their large size remains a bottleneck for deployment on resource-constrained edge devices. This paper proposes a framework to design fine-tuned variants of Whisper which address the above problem. Structured sparsity is enforced via the Sparse Group LASSO penalty as a loss regularizer, to reduce the number of FLOating Point operations (FLOPs). Further, a weight statistics aware pruning algorithm is proposed. We also design our custom text normalizer for WER evaluation. On Common Voice 11.0 Hindi dataset, we obtain, without degrading WER, (a) 35.4% reduction in model parameters, 14.25% lower memory consumption and 18.5% fewer FLOPs on Whisper-small, and (b) 31% reduction in model parameters, 15.29% lower memory consumption and 16.95% fewer FLOPs on Whisper-medium; and, (c) substantially outperform the state-of-the-art Iterative Magnitude Pruning based method by pruning 18.7% more parameters along with a 12.31 reduction in WER.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。