提出新方法加速语音大模型蒸馏，提升部署效率。
Fast Speech Foundation Model Distillation Using Interleaved Stacking
- 采用交错堆叠策略逐步增加模型深度，保持层位置一致
- 在SUPERB上实现与全量训练相当的性能，速度更快
- 适合需要快速部署高效语音模型的低资源场景
将大型语音基础模型（SFM）蒸馏为高效的学生模型已成功应用于低资源环境。尽管蒸馏能降低推理延迟，但仍需额外的学生模型训练。然而，SFM蒸馏的训练效率尚未得到充分探索。本文研究了SFM蒸馏的训练加速，以加快模型部署。我们考察了堆叠方法，即通过训练逐步增加模型深度直至达到目标深度。现有堆叠方法虽提升了训练速度，但存在性能下降问题。为此，我们提出交错堆叠，一种新型堆叠方法，能够在整个堆叠过程中始终保留层的位置信息。这一特性在SFM中尤为重要，因为每层编码特定的层级知识。我们在SUPERB数据集上验证了该方法的有效性。
原文摘要 · Abstract (English)
Distilling a large speech foundation model (SFM) into an efficient student model has been successfully applied to low-resource environments. Although distillation reduces inference latency, it requires an additional student model training. However, the training efficiency of SFM distillation remains underexplored. In this work, we explore training acceleration of SFM distillation to speed up model deployment. We examine the potential of stacking, in which the model depth is progressively increased through training until the target model depth is reached. While existing stacking methods improve training speed, they suffer from performance degradation. To handle this limitation, we propose interleaved stacking, a novel stacking method that consistently preserves layer position throughout the stacking process. This property is particularly critical in SFMs, in which each layer encodes distinct layer-specific knowledge. We validate the effectiveness of the proposed method on SUPERB.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。