对比12种高效微调方法，发现标准LoRA在强化推理中表现不佳。
Evaluating Parameter Efficient Methods for RLVR
- 测试多种结构化变体，如DoRA、AdaLoRA等，优于传统LoRA。
- 发现谱崩溃现象导致基于SVD初始化的方法失效。
- 极低参数量（如Rank-1）严重限制模型推理能力。
我们系统评估了在可验证奖励强化学习（RLVR）范式下参数高效微调（PEFT）方法的表现。尽管LoRA被广泛使用，但其在RLVR中的最优架构仍不明确。本研究首次对超过12种PEFT方法在DeepSeek-R1-Distill系列模型上进行综合评估，覆盖数学推理基准。结果挑战了默认使用标准LoRA的惯例：第一，结构化变体如DoRA、AdaLoRA和MiSS持续优于LoRA；第二，发现基于SVD初始化策略（如PiSSA、MiLoRA）存在谱崩溃现象，主成分更新与强化学习优化目标存在根本性错配；第三，消融实验表明极端参数压缩（如VeRA、Rank-1）严重制约推理能力。通过消融与缩放实验进一步验证结论。本工作为探索更高效的参数高效强化学习方法提供了明确指导。
原文摘要 · Abstract (English)
We systematically evaluate Parameter-Efficient Fine-Tuning (PEFT) methods under the paradigm of Reinforcement Learning with Verifiable Rewards (RLVR). RLVR incentivizes language models to enhance their reasoning capabilities through verifiable feedback; however, while methods like LoRA are commonly used, the optimal PEFT architecture for RLVR remains unidentified. In this work, we conduct the first comprehensive evaluation of over 12 PEFT methodologies across the DeepSeek-R1-Distill families on mathematical reasoning benchmarks. Our empirical results challenge the default adoption of standard LoRA with three main findings. First, we demonstrate that structural variants, such as DoRA, AdaLoRA, and MiSS, consistently outperform LoRA. Second, we uncover a spectral collapse phenomenon in SVD-informed initialization strategies (\textit{e.g.,} PiSSA, MiLoRA), attributing their failure to a fundamental misalignment between principal-component updates and RL optimization. Furthermore, our ablations reveal that extreme parameter reduction (\textit{e.g.,} VeRA, Rank-1) severely bottlenecks reasoning capacity. We further conduct ablation studies and scaling experiments to validate our findings. This work provides a definitive guide for advocating for more exploration for parameter-efficient RL methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。