动态调整惩罚策略,让大模型推理更准更稳。
Adaptive Negative Reinforcement for LLM Reasoning:Dynamically Balancing Correction and Diversity in RLVR

- 根据训练阶段动态调整错误惩罚强度,初期重纠偏,后期微调
- 按模型对错误路径的信心程度加权惩罚,高自信错罚得更重
- 适合需要提升数学推理能力的LLM开发者或研究者
基于可验证奖励的强化学习(RLVR)已成为提升大语言模型(LLMs)推理能力的有效方法。近期研究表明,负样本强化(NSR)——通过惩罚错误步骤而非仅奖励正确结果——在全范围的Pass@k指标上可媲美甚至超越PPO和GRPO等复杂框架。然而现有NSR方法通常在整个训练过程中采用固定惩罚,且对所有错误一视同仁。为此,本文提出两种改进:自适应负样本强化(A-NSR),使用随时间变化的调度函数,初期侧重纠正错误以稳定模型,后期转为更精细的调控;置信度加权负强化(CW-NSR),根据模型对错误路径的归一化序列似然分配不同惩罚权重,高自信错误惩罚更重,低信心探索性错误则轻罚。形式化分析表明,这些机制可控制令牌级更新,利用先验引导的概率重分布,同时自然抵御过拟合。我们在MATH、AIME 2025和AMC23等难题数据集上,使用Qwen2.5-Math-1.5B架构进行了评估。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has become a highly effective method for improving the reasoning abilities of Large Language Models (LLMs). Recent research shows that Negative Sample Reinforcement (NSR) -- which focuses on penalizing incorrect steps rather than simply rewarding correct ones -- can match or even exceed the performance of more complex frameworks like PPO and GRPO across the entire Pass@k spectrum. However, current NSR techniques usually apply a fixed penalty throughout the training process and treat every incorrect response with the same weight. To address these limitations, we propose two extensions to the NSR framework: Adaptive Negative Sample Reinforcement. Rather than using a fixed update rule, A-NSR uses time-dependent scheduling functions. In the initial training phases, the system focuses heavily on correcting errors to stabilize the model. As training continues, it shifts toward more subtle and controlled updates. We also introduce Confidence-Weighted Negative Reinforcement, which operates on the principle that different mistakes carry different levels of importance. CW-NSR assigns specific penalty weights based on the model's normalized sequence likelihood. If the model is highly confident in a wrong path, it receives a larger penalty and for uncertain errors -- where the model is effectively exploring -- are penalized less strictly. Our formal analysis shows how these mechanisms govern token-level updates, allowing the model to leverage prior-guided probability redistribution while providing a natural defense against overfitting. We evaluated these methods on difficult reasoning datasets, including MATH, AIME 2025, and AMC23, using the Qwen2.5-Math-1.5B architecture.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。