提出Q3R正则化，让低秩训练更有效且接近全参数模型性能。
Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training
- 基于二次重加权思想，动态优化权重矩阵的低秩结构。
- 在ViT-Tiny上压缩60%和80%参数，准确率仅下降1.3%和4%。
- 适用于视觉与语言任务，兼容现有架构，计算开销小。
基于低秩优化的参数高效训练已成为微调大模型的成功方法。然而，在低秩预训练中，同时保持低秩权重结构并优化任务目标仍具挑战。本文提出一种新的低秩诱导训练策略——二次重加权秩正则化（Q3R），其灵感来自迭代重加权最小二乘（IRLS）框架。Q3R采用一个二次正则项，以逼近平滑化的对数行列式秩代理函数。与现有方法不同，Q3R可在指定目标秩下训练权重矩阵，实现与密集模型相当的预测性能，且计算开销小、可完全兼容现有架构。例如，在CIFAR-10上，对ViT-Tiny进行Q3R正则化后，参数量分别缩减至60%和80%，准确率仅下降1.3%和4%。我们在视觉与语言任务的Transformer模型上验证了Q3R的有效性，包括低秩微调场景。
原文摘要 · Abstract (English)
Parameter-efficient training based on low-rank optimization has become a highly successful tool for fine-tuning large deep learning models. However, these methods often fail for low-rank pre-training, where simultaneously maintaining low-rank weight structure and optimizing the task objective remains challenging. We propose the $\textit{Quadratic Reweighted Rank Regularizer}$ ($\texttt{Q3R}$), which leads to a novel low-rank-inducing training strategy inspired by the Iteratively Reweighted Least Squares (IRLS) framework. $\texttt{Q3R}$ is based on a quadratic regularizer term that majorizes a smoothed log-determinant rank surrogate. Unlike other low-rank training techniques, $\texttt{Q3R}$ can train weight matrices to prescribed low target ranks while achieving predictive performance comparable to dense models, with small computational overhead and full compatibility with existing architectures. For example, we demonstrate a $\texttt{Q3R}$-regularized ViT-Tiny experiment where truncating the model to $60\%$ and $80\%$ of its parameters results in only minor absolute accuracy drops of $1.3\%$ and $4\%$, respectively, on CIFAR-10. We confirm the efficacy of $\texttt{Q3R}$ on Transformers across both vision and language tasks, including low-rank fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。