改进低秩微调方法，解决高阶时过拟合问题，性能更接近全量微调。
GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning
- 将权重矩阵分块，每块独立低秩适配，打破梯度纠缠
- 在HumanEval+上提升8.5%准确率，优于现有方法
- 适配不同模型规模和秩设置，适合高效微调场景
低秩适配（LoRA）是生成模型参数高效微调（PEFT）的流行方法，因其简洁有效而广受青睐。尽管已有改进，LoRA仍存在根本缺陷：当瓶颈扩大时易过拟合，在秩32-64时表现最佳，更高秩下准确率停滞或下降，仍不及全量微调（FFT）性能。我们发现根源在于LoRA的结构瓶颈，导致无关输入通道间梯度耦合，扭曲梯度传播。为此，提出新型结构——细粒度低秩适配（GraLoRA），将权重矩阵划分为子块，每块配备独立低秩适配器。计算与存储开销几乎不变，成功克服LoRA局限，显著提升表征能力，更接近FFT行为。在代码生成与常识推理基准测试中，GraLoRA持续优于LoRA及其他基线，于HumanEval+上实现最高+8.5%的Pass@1绝对提升。该优势跨模型规模与秩设置保持稳定，展现出色可扩展性与鲁棒性。代码、数据与脚本见https://github.com/SqueezeBits/GraLoRA.git
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) is a popular method for parameter-efficient fine-tuning (PEFT) of generative models, valued for its simplicity and effectiveness. Despite recent enhancements, LoRA still suffers from a fundamental limitation: overfitting when the bottleneck is widened. It performs best at ranks 32-64, yet its accuracy stagnates or declines at higher ranks, still falling short of full fine-tuning (FFT) performance. We identify the root cause as LoRA's structural bottleneck, which introduces gradient entanglement to the unrelated input channels and distorts gradient propagation. To address this, we introduce a novel structure, Granular Low-Rank Adaptation (GraLoRA) that partitions weight matrices into sub-blocks, each with its own low-rank adapter. With negligible computational or storage cost, GraLoRA overcomes LoRA's limitations, effectively increases the representational capacity, and more closely approximates FFT behavior. Experiments on code generation and commonsense reasoning benchmarks show that GraLoRA consistently outperforms LoRA and other baselines, achieving up to +8.5% absolute gain in Pass@1 on HumanEval+. These improvements hold across model sizes and rank settings, making GraLoRA a scalable and robust solution for PEFT. Code, data, and scripts are available at https://github.com/SqueezeBits/GraLoRA.git
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。