arXiv:2605.07366cs.CL2026-05中稿 · EMNLP

LoRA自适应调参在强化学习中失效,因梯度分布平坦且秩分配会放大不平等。

When Gradient Importance Lies: Adaptive LoRA Rank Allocation Fails Under GRPO

论文配图:When Gradient Importance Lies: Adaptive LoRA Rank Allocation Fails Under GRPO
图 1 · 摘自论文原文
  • 基于梯度重要性动态分配参数,但强化学习中效果反而更差。
  • 非均匀分配使梯度差异从2.17倍扩大到3.00倍,导致低秩层被抑制。
  • 适合关注对齐训练中参数效率与模型稳定性的研究者参考。

LoRA的自适应秩分配(为重要层分配更多参数)在监督微调(SFT)中显著提升效率。我们测试其在强化学习中的表现,聚焦于分组相对策略优化(GRPO)。以Qwen 2.5 1.5B模型和GSM8K数据集进行梯度幅值分析发现,在相同参数预算下,按比例分配秩使准确率下降4.5个百分点(70.0% vs. 74.5%),而均匀分配表现更优。原因在于:GRPO下的梯度分布远比SFT平坦,最大最小层重要性比仅为2.17倍,远低于SFT中超过10倍的差距;所有层均承载有意义梯度,无“闲置”层。此外,非均匀分配引发梯度放大效应,使重要性差异从2.17倍扩大至3.00倍,形成正反馈循环:高秩层吸收更多梯度,低秩层逐渐被抑制。随机分配控制实验显示秩与梯度占比相关性达r=0.972,表明秩决定了重要性而非相反。该结果为单种子、单任务,提示梯度重要性未必能预测强化学习中的容量需求,应谨慎将SFT阶段的秩分配策略迁移至对齐训练。

原文摘要 · Abstract (English)

Adaptive rank allocation for LoRA - allocating more parameters to important layers and fewer to unimportant ones - consistently improves efficiency under supervised fine-tuning (SFT). We test whether this success transfers to reinforcement learning, specifically Group Relative Policy Optimization (GRPO). Using gradient-magnitude profiling on Qwen 2.5 1.5B with GSM8K, we find that, in our setting, it does not: proportional rank allocation degrades accuracy by 4.5 points compared to uniform allocation (70.0% vs. 74.5%), despite using identical parameter budgets. We identify two mechanisms behind this failure. First, the gradient landscape under GRPO is fundamentally flatter than under SFT: the max-to-min layer importance ratio is only 2.17x, whereas the layer concentration reported by Shi et al. (2024) for SFT (top 30% of layers carrying >80% of the gradient signal) implies a max/min ratio well above 10x. All layers carry meaningful gradient signal; none are truly idle. Second, we observe a gradient amplification effect: non-uniform allocation widens the importance spread from 2.17x to 3.00x, creating a positive feedback loop where high-rank layers absorb more gradient while low-rank layers are progressively silenced. A random-allocation control yields the same amplification (r=0.972 correlation between assigned rank and resulting gradient share), indicating that rank causally determines gradient importance rather than the reverse. The negative result is single-seed and single-task; we present it as preliminary evidence that gradient importance does not predict capacity requirements under RL, and that naive transfer of SFT-era rank allocation strategies to alignment training should be evaluated cautiously.

Lo适配强化学习参数效率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。