arXiv:2608.09324cs.AI2026-08

用图模型替代投票,让测试时强化学习更精准地奖励合理推理。

CoRE: Consensus Rewards via Equilibrium for Test-Time Reinforcement Learning

论文配图:CoRE: Consensus Rewards via Equilibrium for Test-Time Reinforcement Learning
图 1 · 摘自论文原文
  • 将多个推理路径建模为带权重的图,通过动态机制提取共识
  • 平均提升21.7分,比传统投票方法高出1.3分,且收敛更快
  • 适合需要高可靠性推理的复杂任务,如数学与逻辑题

在无标注测试数据上,强化学习缺乏真实奖励信号;现有测试时强化学习方法通过模型自身采样结果进行投票,仅奖励多数答案,忽视少数正确答案,并对所有多数匹配结果给予相同评分。本文提出CoRE(基于均衡的共识奖励):将N个采样轨迹构建成图,边权融合答案一致性、推理相似度与生成置信度,利用复制者动力学提取主导集合,生成精细化伪标签、逐轨迹评分及问题级连贯性门控。CoRE严格推广了投票机制:多数投票是其特例;块值分析给出在少数正确但多数错误时恢复正确性的临界阈值;置信度校准可使该阈值乘法降低。在七个骨干模型与五个基准测试中(42个模型-基准组合,每组三随机种子),CoRE相较未训练基线平均提升21.7分,优于多数投票方法的20.4分,在分歧场景下最大领先达+7.5分,且在54%–70%更少步数内达到投票基线的准确率上限。共识不计数:将轨迹群体视为图而非选票箱,以零额外采样成本实现可校准、分级、自监督的奖励机制。

原文摘要 · Abstract (English)

On unlabeled test data, reinforcement learning lacks a ground-truth reward; test-time RL methods derive one from the model's own roll-outs, rewarding those that match the majority vote over $N$ sampled answers. That vote discards a correct answer whenever it is a minority and scores every majority-matching roll-out identically. We replace it with \emph{CoRE} (Consensus Rewards via Equilibrium): the $N$ roll-outs form a graph whose edges combine answer agreement, reasoning similarity, and generation confidence, and replicator dynamics extract its dominant set, yielding a refined pseudo-label, a graded per-roll-out reward, and a per-question cohesiveness gate. CoRE strictly generalizes voting: majority voting is recovered as a special case; a block-value analysis gives a sharp threshold for when consensus recovers a correct minority against a larger wrong plurality; and confidence calibration provably lowers that threshold multiplicatively. Across seven backbones and five benchmarks (42 model--benchmark cells, three seeds each), \emph{CoRE} improves the untrained base by $+21.7$ points on average versus $+20.4$ for majority-vote TTRL, wins wherever agreement is contestable with margins over the vote of up to $+7.5$ points, and reaches the voting baseline's plateau accuracy in $54$--$70$\% fewer steps. Consensus, not counting: treating the roll-out group as a graph rather than a ballot box turns a brittle vote into a calibrated, graded, self-supervised reward at no extra roll-out cost.

强化学习测试时学习共识机制自监督

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。