针对大模型推理的强化学习，提出自适应策略优化方法，提升难易适中问题的学习效率。
The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoning
- 根据问题难度和模型信心动态调整更新权重
- 在12个基准上显著优于现有强化学习方法
- 适合需要高效利用计算资源的模型训练场景
基于可验证奖励的强化学习能显著提升大模型推理能力，但传统GRPO式训练对简单、困难和可学问题采用统一采样与加权，导致计算资源分配低效。我们通过追踪分词对数概率、组归一化优势和逐标记更新权重，揭示了训练过程中三个持续出现的现象：(1) 信心膨胀，(2) 优势收缩，(3) 分层收敛。这些发现表明每次更新的价值强烈依赖于问题难度与模型当前能力。受此启发，我们提出自信与难度自适应策略优化（CoDaPO），为每个问题赋予基于回溯信心与经验难度的有限值，据此重加权策略更新，并在小批量内重新采样高价值可学问题，从而在固定计算预算下提升可学区间内的探索效率。在十二个基准上，CoDaPO始终优于现有强化学习方法。代码已公开于 https://github.com/tmlr-group/CoDaPO。
原文摘要 · Abstract (English)
RL with verifiable rewards can substantially improve LLM reasoning, yet standard GRPO-style training often treats easy, hard, and learnable questions alike through uniform sampling and weighting, leading to inefficient compute allocation. We study GRPO by tracking token log-probabilities, group-normalized advantages, and the induced token-level update weights. This reveals three recurring dynamics as training proceeds: (1) confidence inflation, (2) advantage contraction, and (3) hierarchical convergence. These findings suggest that the utility of each update depends strongly on both question difficulty and the model's current competence. Motivated by this, we propose Confidence and Difficulty-adaptive Policy Optimization (CoDaPO), which assigns each question a bounded value from rollout confidence and empirical difficulty. CoDaPO then uses this value to reweight policy updates and resample high-value learnable questions within mini-batches, thereby increasing discovery within the learnable band under a fixed compute budget. Across twelve benchmarks, CoDaPO consistently improves accuracy over existing RL methods. Our code is publicly available at https://github.com/tmlr-group/CoDaPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。