用正交Softmax高效筛选语音模型子网，一次训练适配多种硬件。
Efficient Supernet Training with Orthogonal Softmax for Scalable ASR Model Compression
- 提出OrthoSoftmax，通过正交Softmax快速定位最优子网。
- 相同训练量下，各尺寸模型的错误率优于或接近独立训练结果。
- 支持按计算量等多维度灵活选型，适合部署于多样硬件环境。
语音识别系统需适配不同硬件环境，本文采用超网络训练联合优化多个不同规模的编码器，实现无需重复训练即可动态调整模型大小以匹配硬件约束。同时提出一种新方法OrthoSoftmax，通过多个正交Softmax函数高效识别超网络中的最优子网，避免高成本搜索。该方法还支持基于多种标准和粒度的灵活精确选择。在Librispeech和TED-LIUM-v2上使用CTC的实验表明，基于FLOPs的组件级选择取得最佳整体性能；相同训练次数下，所有模型尺寸的词错误率（WER）均达到或略优于独立训练模型。进一步分析揭示了所选组件的规律性特征。
原文摘要 · Abstract (English)
ASR systems are deployed across diverse environments, each with specific hardware constraints. We use supernet training to jointly train multiple encoders of varying sizes, enabling dynamic model size adjustment to fit hardware constraints without redundant training. Moreover, we introduce a novel method called OrthoSoftmax, which applies multiple orthogonal softmax functions to efficiently identify optimal subnets within the supernet, avoiding resource-intensive search. This approach also enables more flexible and precise subnet selection by allowing selection based on various criteria and levels of granularity. Our results with CTC on Librispeech and TED-LIUM-v2 show that FLOPs-aware component-wise selection achieves the best overall performance. With the same number of training updates from one single job, WERs for all model sizes are comparable to or slightly better than those of individually trained models. Furthermore, we analyze patterns in the selected components and reveal interesting insights.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。