用小模型反复展开，实现语音模型高效压缩且不降性能
Unfolding A Few Structures for The Many: Memory-Efficient Compression of Conformer and Speech Foundation Models
- 用少量模块训练小模型，再多次展开模拟大模型
- 参数减少35%（Conformer）和30%（wav2vec2），性能不变
- 适合资源受限场景的语音模型部署
本文提出一种面向 Conformer 语音识别与语音基础模型的新型内存高效压缩方法。该方法采用独特的‘小到大’设计：先训练一个仅含少数 Conformer 或 Transformer 模块的紧凑‘种子’模型，再在单次展开周期内将其多次展开，以模拟不同逻辑深度的大模型表现。种子模型与多个展开路径联合训练，并通过最大展开模型与最小种子模型之间的 KL 散度进行自蒸馏，降低性能差距。实验表明，该可折叠模型在多种深度配置下，语音识别性能可媲美独立构建的 Conformer 与 wav2vec2/HuBERT 语音基础模型，同时仅需极低内存与存储。Conformer 和 wav2vec2 模型分别实现 35% 和 30% 的参数压缩，性能无损。
原文摘要 · Abstract (English)
This paper presents a novel memory-efficient model compression approach for Conformer ASR and speech foundation systems. Our approach features a unique "small-to-large" design. A compact "seed" model containing a few Conformer or Transformer blocks is trained and unfolded many times to emulate the performance of larger uncompressed models with different logical depths. The seed model and many unfolded paths are jointly trained within a single unfolding cycle. The KL-divergence between the largest unfolded and smallest seed models is used in a self-distillation process to minimize their performance disparity. Experimental results show that our foldable model produces ASR performance comparable to individually constructed Conformer and wav2vec2/HuBERT speech foundation models under various depth configurations, while requiring only minimal memory and storage. Conformer and wav2vec2 models with a reduction of 35% and 30% parameters are obtained without loss of performance, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。