针对注意力机制提出二级度量剪枝法，大幅减少参数量仍保持性能。
The silence of the weights: a structural pruning strategy for attention-based audio signal architectures with second order metrics
- 按头和四层分别剪枝，用二阶度量评估参数重要性。
- 注意力模块剪掉50%参数后，音频模型性能基本不变。
- 适合需要轻量化部署的语音处理场景，如边缘设备。
基于Transformer的模型在自然语言处理与机器听觉等领域已成主流，得益于注意力机制。然而，注意力层参数量大，训练与推理需高端硬件支持。本文提出一种专为注意力机制设计的新通道剪枝技术，将每个注意力头及查询、键、值、输出投影四层独立剪枝，并采用二阶度量评估网络参数重要性。对比了头剪枝与基于幅值的评分策略，在Audio Spectrogram Transformer (AST) 和 Whisper 模型上验证效果。结果表明，即使剪掉注意力模块50%的参数，模型性能仍基本保持。
原文摘要 · Abstract (English)
Transformer-based models have become the state of the art across multiple domains, from natural language processing to machine listening, thanks to the attention mechanisms. However, the attention layers require a large number of parameters and high-end hardware for both training and inference. We propose a novel channel-pruning technique explicitly targeted at the attention mechanism, decoupling the pruning of each head and the four layers in the attention block: query, key, value, and output projection matrices, employing a second-order metric to score the network's parameters. We compare our technique against head-pruning strategies and magnitude-driven scoring metrics, investigating the effects of pruning on Audio Spectrogram Transformer (AST) and Whisper. Our results show that even after pruning 50\% of the parameters in the attention block, performance is largely preserved.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。