提出轻量级训练时低秩正则化方法，提升模型压缩性且几乎不增加训练开销。
SLORR: Simple and Efficient In-Training Low-Rank Regularization
- 直接对原始权重矩阵施加低秩正则，无需修改架构或引入额外参数。
- 在图像和大模型上测试，压缩后性能损失小，训练开销低于8%（图像）和1%（大模型）。
- 适合追求高效模型压缩的开发者，尤其适用于大规模预训练场景。
低秩分解广泛用于压缩神经网络，但现代模型通常难以在不显著损失精度的情况下进行激进分解。现有训练时低秩正则化方法常需对大型权重矩阵执行SVD、修改模型架构（引入额外可训练参数）或依赖有状态缓存数据。为解决这些问题，我们提出SLORR——一种简单、无状态、保持原架构的训练时低秩正则化框架，基于霍尔指数稀疏度与核范数设计两种主要变体。SLORR通过GPU友好的近似方法实现正则项前向与反向传播，并提供近似保证。我们在ImageNet-1K上评估了短周期持续训练的ResNet-50、ViT-B/16、ViT-L/16以及ResNet-18的预训练任务，结果显示SLORR在引入少于8%训练开销的同时提升了模型可压缩性。进一步在135M和560M规模的大语言模型预训练中评估SLORR-Hoyer，结果表明其压缩模型性能保留优于未正则化模型，平均训练开销不足1%。
原文摘要 · Abstract (English)
Low-rank factorization is widely used to compress neural networks, but modern models are often not naturally amenable to aggressive factorization without significant accuracy loss. Existing training-time low-rank regularizers can improve compressibility, but they often require SVDs of large weight matrices, modify the model architecture (introducing additional trainable parameters), or rely on stateful cached quantities. To address these limitations, we introduce SLORR, a simple, stateless, and architecture-preserving framework for in-training low-rank regularization, instantiated with two main variants based on the Hoyer sparsity metric and the nuclear norm. SLORR directly regularizes the original weight matrices using GPU-friendly approximations for the forward and backward passes of the regularizers, for which we provide approximation guarantees. We first evaluate SLORR on ImageNet-1K across short-horizon continued training of ResNet-50, ViT-B/16, and ViT-L/16, and pretraining of ResNet-18, where SLORR induces compressibility while introducing less than 8% training overhead. We further evaluate SLORR-Hoyer in LLM pretraining at 135M and 560M scales: SLORR-trained compressed models preserve performance substantially better than unregularized models while adding less than 1% average training overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。