用自适应丢弃法在训练中同时剪枝，显著减少参数量并提升语音识别准确率。
Adaptive Dropout for Pruning Conformers
- 基于单元级保留概率的自适应丢弃，通过反向传播与Gumbel-Softmax估算可剪枝单元
- 在Conformer的三个位置引入该方法，参数量减少54%，词错误率降低约1%
- 适合需要轻量化模型且追求高精度的语音识别应用
本文提出一种基于单元级保留概率的自适应丢弃方法，实现训练与剪枝的联合优化。通过反向传播和Gumbel-Softmax技术估算每个单元的保留概率，低保留概率的单元被视为可剪枝。该方法在Conformer块的三个位置应用：(a) 前馈网络隐藏层，(b) 自注意力中的查询与值向量，(c) LConv组件输入向量。在LibriSpeech语音识别任务上的实验表明，该方法在参数量减少54%的同时，词错误率（WER）降低约1%。结果证明该方法能有效实现模型压缩与性能提升的双重目标。
原文摘要 · Abstract (English)
This paper proposes a method to effectively perform joint training-and-pruning based on adaptive dropout layers with unit-wise retention probabilities. The proposed method is based on the estimation of a unit-wise retention probability in a dropout layer. A unit that is estimated to have a small retention probability can be considered to be prunable. The retention probability of the unit is estimated using back-propagation and the Gumbel-Softmax technique. This pruning method is applied at several application points in Conformers such that the effective number of parameters can be significantly reduced. Specifically, adaptive dropout layers are introduced in three locations in each Conformer block: (a) the hidden layer of the feed-forward-net component, (b) the query vectors and the value vectors of the self-attention component, and (c) the input vectors of the LConv component. The proposed method is evaluated by conducting a speech recognition experiment on the LibriSpeech task. It was shown that this approach could simultaneously achieve a parameter reduction and accuracy improvement. The word error rates improved by approx 1% while reducing the number of parameters by 54%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。