让大模型推理更聪明：用新方法避免答案千篇一律
Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data
- 用均匀采样打破模型偏见，保持解题多样性
- 在AIME25上提升15.1%准确率，显著改善泛化能力
- 适合追求高质量推理的AI研究者和开发者
强化学习（RL）能提升大模型的推理能力，但随着模型规模增大，基础模型在标准基准（如MATH）上趋于饱和，导致正确但单一的答案。这种环境下，组内相对算法（如GRPO）因缺乏失败案例而失去优势信号，引发策略模式崩溃。为此，我们提出无参解码策略CUTS，通过从高置信度候选中均匀采样，保持结构并促进探索。将其集成到Mixed-CUTS训练框架中，结合利用与探索的轨迹，增强组内优势方差。在Qwen3模型上的实验表明，该方法有效防止策略退化，并显著提升跨域泛化性能。特别地，Mixed-CUTS在具有挑战性的AIME25基准上，相比标准GRPO将Pass@1准确率提升最高达15.1%，验证了维持语义流形内多样性对严谨推理至关重要。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) enhances LLM reasoning, yet a paradox emerges as models scale: strong base models saturate standard benchmarks (e.g., MATH), yielding correct but homogeneous solutions. In such environments, the lack of failure cases causes the advantage signal in group-relative algorithms (e.g., GRPO) to vanish, driving policies into mode collapse. To address this, we propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration. Unlike standard sampling that follows model biases, CUTS flattens the local optimization landscape by sampling uniformly from constrained high-confidence candidates. We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance. Experiments on Qwen3 models demonstrate that our approach prevents policy degeneration and significantly boosts out-of-domain generalization. Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO, validating that maintaining diversity within the semantic manifold is critical for rigorous reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。