通过分解权重矩阵提升LoRA训练稳定性和知识迁移效率
Dual Decomposition of Weights and Singular Value Low Rank Adaptation
- 将权重矩阵分解为大小与方向，用SVD进行合理初始化
- 在MMLU上达48.35%准确率，在GSM8K上达62.53%（±1.59）
- 适合需要精准知识迁移的领域任务，理论与实证兼备
参数高效微调（PEFT）已成为适配大语言模型（LLMs）至下游任务的关键范式，其中低秩适配（LoRA）是最广泛采用的方法之一。然而，现有基于LoRA的方法存在两个根本缺陷：训练动态不稳定和从预训练模型中知识迁移效率低，根源在于适配器参数的随机初始化。为此，我们提出DuDe，一种新方法，将权重矩阵分解为幅度与方向成分，利用奇异值分解（SVD）实现有原则的初始化。全面评估表明，DuDe在性能与鲁棒性方面均表现优异，在MMLU上达到最高48.35%准确率，在GSM8K上达62.53%（±1.59）。理论分析与实证验证共同证明，该分解策略提升了优化稳定性，并更好地保留了预训练表示，尤其适用于需特定领域知识的任务。结合稳健的实证表现与严谨的理论基础，DuDe成为LLM PEFT方法的重要贡献。
原文摘要 · Abstract (English)
Parameter-Efficient Fine-Tuning (PEFT) has emerged as a critical paradigm for adapting Large Language Models (LLMs) to downstream tasks, among which Low-rank Adaptation (LoRA) represents one of the most widely adopted methodologies. However, existing LoRA-based approaches exhibit two fundamental limitations: unstable training dynamics and inefficient knowledge transfer from pre-trained models, both stemming from random initialization of adapter parameters. To overcome these challenges, we propose DuDe, a novel approach that decomposes weight matrices into magnitude and direction components, employing Singular Value Decomposition (SVD) for principled initialization. Our comprehensive evaluation demonstrates DuDe's superior performance and robustness, achieving up to 48.35\% accuracy on MMLU and 62.53\% ($\pm$ 1.59) accuracy on GSM8K. Our theoretical analysis and empirical validation collectively demonstrate that DuDe's decomposition strategy enhances optimization stability and better preserves pre-trained representations, particularly for domain-specific tasks requiring specialized knowledge. The combination of robust empirical performance and rigorous theoretical foundations establishes DuDe as a significant contribution to PEFT methodologies for LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。