用推理质量提升示范效果，让好思路教好模型。
Good Reasoning Makes Good Demonstrations: Implicit Reasoning Quality Supervision via In-Context Reinforcement Learning
- 让模型自己评估示范质量，通过上下文学习捕捉推理优劣
- 实验显示准确率与推理质量双提升，优于传统强化学习方法
- 适合想提升模型逻辑严谨性的研究者和开发者
基于可验证奖励的强化学习（RLVR）能改善大模型的推理能力，但对所有正确答案一视同仁，可能强化偶然得出正确答案的错误推理路径。我们发现：更好的推理会产生更有效的示范——高质量解法作为上下文示例比低质量解法更具教学价值。我们称此为「示范效用」，并提出利用策略模型自身的上下文学习能力来衡量它，得到一个称为「证据增益」的质量信号。为在训练中使用该信号，我们引入「上下文强化学习」（In-Context RLVR），在每次推理前添加示范。理论上证明，这种输入修改可隐式按证据增益比例重加权奖励，显著提升高质量推理路径的权重，无需额外计算开销。在数学推理基准测试中，该方法持续提升准确率与推理质量，优于标准RLVR基线。代码与数据集见https://github.com/Mithas-114/IC-DAPO。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in large language models but treats all correct solutions equally, potentially reinforcing flawed traces that arrive at correct answers by chance. We observe that \emph{better reasoning makes better demonstrations}: high-quality solutions serve as more effective in-context examples than low-quality ones. We term this teaching ability \textbf{Demonstration Utility}, and show that the policy model's own in-context learning ability provides an efficient way to measure it, yielding a quality signal termed \textbf{Evidence Gain}. To leverage this signal during training, we introduce \textbf{In-Context RLVR}, which prepends demonstrations before each rollout. Theoretically, we prove that this simple input modification implicitly reweights rewards by a factor approximately proportional to Evidence Gain, assigning higher weights to high-quality traces without requiring costly computation. Experiments on mathematical reasoning benchmarks demonstrate consistent improvements in both accuracy and reasoning quality over standard RLVR baselines. Our codes and datasets are available at https://github.com/Mithas-114/IC-DAPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。