低秩训练让大模型更高效，语音识别性能不降反升
Full-Rank No More: Low-Rank Weight Training for Modern Speech Recognition Models
- 仅对注意力模块用低秩结构，降幅12%仍能提升性能
- 前馈层降50%秩就性能下降，需精细设计
- 结合SVD初始化和分层秩分配，训练效率提1.3倍
本文研究大规模基于Conformer的语音识别模型从零开始进行低秩权重训练的可行性。结果表明，仅在注意力模块应用低秩结构即可意外提升性能，即使秩降低12%；而前馈层则更为敏感，秩降低50%即出现性能下降。此外，初始化方式与逐层秩分配对训练成功至关重要：采用SVD初始化和线性层间秩映射可显著提升效果。基于此，我们提出低秩语音模型从零训练（LR-SMS），在保持全秩训练性能的同时，参数量减少至少2倍，训练速度提升1.3倍（语音识别）和1.15倍（视听语音识别）。
原文摘要 · Abstract (English)
This paper investigates the under-explored area of low-rank weight training for large-scale Conformer-based speech recognition models from scratch. Our study demonstrates the viability of this training paradigm for such models, yielding several notable findings. Firstly, we discover that applying a low-rank structure exclusively to the attention modules can unexpectedly enhance performance, even with a significant rank reduction of 12%. In contrast, feed-forward layers present greater challenges, as they begin to exhibit performance degradation with a moderate 50% rank reduction. Furthermore, we find that both initialization and layer-wise rank assignment play critical roles in successful low-rank training. Specifically, employing SVD initialization and linear layer-wise rank mapping significantly boosts the efficacy of low-rank weight training. Building on these insights, we introduce the Low-Rank Speech Model from Scratch (LR-SMS), an approach that achieves performance parity with full-rank training while delivering substantial reductions in parameters count (by at least 2x), and training time speedups (by 1.3x for ASR and 1.15x for AVSR).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。