提出统一框架,让小模型高效适配大模型,效果超LoRA和DoRA。
Calibrating and Rotating: A Unified Framework for Weight Conditioning in PEFT
- 将DoRA重构成更高效的矩阵形式,揭示其本质是可学习的权重调节
- 设计预对角校准与正交旋转两种新方法,提升更新分布均匀性
- 适合想用小参数高效微调大模型的研究者或工程落地场景
参数高效微调(PEFT)在适应大模型中至关重要,其中LoRA是基础方法。DoRA通过将权重更新分解为幅度与方向,显著提升性能,但其机制不明且计算开销大。本文发现,DoRA的成功源于其提升了权重更新矩阵的奇异值熵,使更新分布更均匀,接近全量微调。我们将其重构为数学等价且更高效的矩阵形式,揭示其本质为可学习的权重调节。基于此,提出统一框架,探索条件矩阵的架构位置与变换类型两个正交维度。在此框架下,提出两种新方法:(1) Pre-Diag,在LoRA更新前应用对角条件矩阵,高效校准预训练权重,提升性能并缩短训练时间;(2) SORA(Skewed Orthogonal Rotation Adaptation),以参数高效方式实现保范正交旋转,更强地变换特征空间。大量实验表明,所提方法在自然语言理解与生成任务上优于LoRA与DoRA,兼具高性能与高效率。代码已开源:https://github.com/MaeChd/SORA。
原文摘要 · Abstract (English)
Parameter-Efficient Fine-Tuning (PEFT) methods are crucial for adapting large pre-trained models. Among these, LoRA is considered a foundational approach. Building on this, the influential DoRA method enhances performance by decomposing weight updates into magnitude and direction. However, its underlying mechanism remains unclear, and it introduces significant computational overhead. In this work, we first identify that DoRA's success stems from its capacity to increase the singular value entropy of the weight update matrix, which promotes a more uniform update distribution akin to full fine-tuning. We then reformulate DoRA into a mathematically equivalent and more efficient matrix form, revealing it as a learnable weight conditioning method. Based on this insight, we propose a unified framework for designing advanced PEFT methods by exploring two orthogonal dimensions: the architectural placement and the transformation type of the conditioning matrix. Within this framework, we introduce two novel methods: (1) \textbf{Pre-Diag}, which applies a diagonal conditioning matrix before the LoRA update to efficiently calibrate the pre-trained weights, thereby enhancing performance while reducing training time; and (2) \textbf{S}kewed \textbf{O}rthogonal \textbf{R}otation \textbf{A}daptation (\textbf{SORA}), which employs a parameter-efficient orthogonal rotation to perform a more powerful, norm-preserving transformation of the feature space. Extensive experiments on natural language understanding and generation tasks demonstrate that our proposed methods achieve superior performance and efficiency compared to both LoRA and DoRA. The code is available at https://github.com/MaeChd/SORA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。