通过投影残差增强大模型推理，兼顾准确率与生成多样性。
ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning
- 将负样本隐藏层投影到正样本低秩子空间，用残差调节梯度。
- 在12个基准上平均表现超越基线，数学推理提升9.4%和7.0%。
- 适合需要高精度且多样输出的复杂任务如数学证明、代码生成。
基于可验证奖励的强化学习（RLVR）虽能提升大语言模型（LLM）的推理能力，但常因过度奖励正样本导致生成多样性不足。尽管负样本强化（NSR）通过加重负样本惩罚缓解此问题，却可能抑制正负响应间的语义共性分布。为此，本文提出负样本投影残差强化学习（ResRL），通过解耦正负响应间相似的语义分布来同时提升推理能力与多样性。理论分析揭示了懒惰似然位移（LLD）与正负头梯度干扰的关系，并推导出单前向代理以界定向量对齐，指导保守优势重加权。ResRL将负样本令牌的隐藏表示投影至基于SVD的低秩正样本子空间，利用投影残差调节负梯度，从而在保持多样性的同时显著提升推理性能，在涵盖数学、代码、智能体任务及函数调用的十二项基准上平均表现优于强基线。特别地，数学推理任务中，ResRL在Avg@16上领先NSR 9.4%，Pass@128上领先7.0%。代码已开源。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) enhances reasoning of Large Language Models (LLMs) but usually exhibits limited generation diversity due to the over-incentivization of positive rewards. Although methods like Negative Sample Reinforcement (NSR) mitigate this issue by upweighting penalty from negative samples, they may suppress the semantic distributions shared between positive and negative responses. To boost reasoning ability without losing diversity, this paper proposes negative sample projection Residual Reinforcement Learning (ResRL) that decouples similar semantic distributions among positive and negative responses. We theoretically link Lazy Likelihood Displacement (LLD) to negative-positive head-gradient interference and derive a single-forward proxy that upper-bounds representation alignment to guide conservative advantage reweighting. ResRL then projects negative-token hidden representations onto an SVD-based low-rank positive subspace and uses projection residuals to modulate negative gradients, improving reasoning while preserving diversity and outperforming strong baselines on average across twelve benchmarks spanning Mathematics, Code, Agent Tasks, and Function Calling. Notably, ResRL surpasses NSR on mathematical reasoning by 9.4\% in Avg@16 and 7.0\% in Pass@128. Code is available at https://github.com/1229095296/ResRL.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。