用多个模型分歧奖励,让挑战者发现真正难题,提升推理能力。
Beyond Uncertainty: Multi-Solver Disagreement Rewards for Self-Evolving Reasoning Curricula

- 用不同能力与采样温度的模型组合作为异议信号
- 在数学竞赛集上平均提分1.34点,效果显著
- 无需改框架、不需额外数据,直接替换奖励函数
自演化推理框架通过训练挑战者生成暴露求解器弱点的问题,构建无需人工数据的自适应课程。但现有方法仅以单个求解器的采样不确定性作为奖励,导致当求解器对挑战者问题分布越来越自信时,所有答案趋于一致,奖励归零,挑战者失去学习信号。更重要的是,单一模型奖励无法区分真正简单的题目与仅符合某求解器偏好的题目。本文提出一种基于异构集成的多求解器分歧奖励,通过计算模型组在每道题上的答案多样性(归一化香农熵),明确奖励那些引发求解器冲突的问题——将难度定义为模型间的差异,而非单模型内部采样方差。该更丰富的梯度使挑战者能发现真正的能力边界问题,生成迫使下游求解器发展跨类型泛化推理策略的课程。该方法为即插即用的奖励替换,无需修改框架或引入额外数据。Qwen3-4B实验显示,在MATH-500、AMC、奥赛等竞赛数学基准上,使用分歧奖励训练的求解器平均提升1.34分,证明多求解器分歧可为自洽推理系统提供互补且可扩展的课程生成信号。
原文摘要 · Abstract (English)
Self-evolving reasoning frameworks train a Challenger to generate questions exposing a Solver's weaknesses, creating adaptive curricula without human data. However, existing approaches use a single solver's sampling uncertainty as the Challenger's reward. This creates a fundamental bottleneck: as the solver grows confident on the Challenger's question distribution, all sampled answers converge identically, collapsing the reward to zero and starving the Challenger of learning signal. Critically, this single-model reward cannot distinguish genuinely easy questions from those that merely align with one solver's learned biases. We propose a multi-solver disagreement reward using a heterogeneous ensemble varying in model capacity and sampling temperature. A normalized Shannon entropy over the ensemble's per-question plurality answers explicitly rewards questions where solvers produce conflicting solutions---capturing difficulty as inter-model divergence rather than intra-model sampling variance. This richer gradient enables the Challenger to discover questions targeting true capability boundaries, producing a curriculum that forces downstream Solvers to develop robust reasoning strategies generalizing across problem types. Our approach is a drop-in reward function replacement requiring no framework modifications or additional data. Experiments with Qwen3-4B show that Solvers trained on disagreement-Challenger questions achieve +1.34 points average improvement on competition-math benchmarks (MATH-500, AMC, Olympiad), suggesting that multi-solver disagreement provides a complementary and scalable signal for curriculum generation in self-play reasoning systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。