针对强化学习中过度自信错误无法纠正的问题，提出不对称惩罚机制提升数学推理模型性能。
Overconfident Errors Need Stronger Correction: Asymmetric Confidence Penalties for Reinforcement Learning
- 根据每条推理路径的置信度差异动态调整惩罚强度，识别并抑制过度自信错误。
- 在三个大模型和两个数学数据集上，全量Pass@k指标均显著提升，尤其改善低概率路径表现。
- 适合关注大模型推理可靠性、强化学习中错误修正机制的研究者与工程师。
基于可验证奖励的强化学习（RLVR）已成为提升大语言模型推理能力的主流范式。然而，现有标准方法存在明显缺陷：虽通过锐化采样提升Pass@1准确率，却同时缩小模型推理边界、降低生成多样性。我们发现其根源在于对错误的统一惩罚——无论数据筛选还是优势归一化方法，均同等对待同一组内的所有错误轨迹。这种同质化处理使被强化的过度自信错误持续占据概率质量，压制有效探索路径。为此，我们提出不对称置信度误差惩罚（ACE），引入每条轨迹的置信度偏移度量c_i = log(pi_theta(y_i|x) / pi_ref(y_i|x))，动态调节负优势。理论证明，ACE梯度可分解为仅作用于过度自信错误的有选择性正则项加上一个可控残差项。我们在VERL框架下，使用GRPO和DAPO对Qwen2.5-Math-7B、Qwen3-8B-Base、Llama-3.1-8B-Instruct在DAPO-Math-17K上微调，评估在MATH-500和AIME 2025上的表现。结果表明，ACE可无缝集成，且在所有模型与基准上稳定提升全量Pass@k指标。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has become the leading paradigm for enhancing reasoning in Large Language Models (LLMs). However, standard RLVR algorithms suffer from a well-documented pathology: while they improve Pass@1 accuracy through sharpened sampling, they simultaneously narrow the model's reasoning boundary and reduce generation diversity. We identify a root cause that existing methods overlook: the uniform penalization of errors. Current approaches -- whether data-filtering methods that select prompts by difficulty, or advantage normalization schemes -- treat all incorrect rollouts within a group identically. We show that this uniformity allows overconfident errors (incorrect reasoning paths that the RL process has spuriously reinforced) to persist and monopolize probability mass, ultimately suppressing valid exploratory trajectories. To address this, we propose the Asymmetric Confidence-aware Error Penalty (ACE). ACE introduces a per-rollout confidence shift metric, c_i = log(pi_theta(y_i|x) / pi_ref(y_i|x)), to dynamically modulate negative advantages. Theoretically, we demonstrate that ACE's gradient can be decomposed into the gradient of a selective regularizer restricted to overconfident errors, plus a well-characterized residual that partially moderates the regularizer's strength. We conduct extensive experiments fine-tuning Qwen2.5-Math-7B, Qwen3-8B-Base, and Llama-3.1-8B-Instruct on the DAPO-Math-17K dataset using GRPO and DAPO within the VERL framework. Evaluated on MATH-500 and AIME 2025, ACE composes seamlessly with existing methods and consistently improves the full Pass@k spectrum across all three model families and benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。