arXiv:2607.07690cs.LGcs.AI2026-07

让两个模型互为考官,通过竞争自动评估推理过程。

Agon: Competitive Cross-Model RL with Implicit Rival Grading of Reasoning

  • 双模型交替扮演出题与解题角色,以击败对方为目标进行训练。
  • 在DeepMath上使pass@1提升至GRPO的两倍,效果远超单一模型。
  • 无需人工标注推理过程,适合构建更可信的复杂推理系统。

基于可验证奖励的强化学习(如GRPO)是当前推理模型的核心驱动力,但仅对最终答案评分。在难题上,这会诱导模型多写而非深思,因为推理过程从未被评价,也无良好思维的标签。我们提出Agon,让两个竞争模型互为评判者:双方解决同一问题,轮流扮演起草与阅读者角色,各自根据超越对方的能力获得奖励。要取胜,模型必须在看到对方工作后仍能更好推理,从而实现推理过程的隐式评估,无需过程标签或奖励模型。由于双方同时优化,每个模型面对的是不断变强的对手,这是单模型强化学习无法实现的。两模型只需能力相近且行为不同。推理时采用双阶段级联结构:先由一模型起草,另一模型阅读后作答。在Qwen3上的DeepMath硬集测试中,该方法将GRPO的pass@1提升一倍,约是未训练混合代理方案的八倍增益。该优势在编程竞赛代码任务及不同模型族(Qwen3.5、Gemma 4)中均复现。目前模型以文本交互;下一步将探索在潜在空间中协同推理。

原文摘要 · Abstract (English)

Reinforcement learning from verifiable rewards (e.g. GRPO) is the engine behind today's reasoning models, yet it grades only the final answer. On hard problems this trains models to write more rather than to think better, since the trace itself is never graded and no label for good thinking exists. We introduce Agon, which makes two competing models each other's graders. Both attempt the same problem; in alternating roles, one drafts a solution and the other reads it while solving, and each is rewarded for out-solving the other. To win, a model must out-reason a rival that has seen its work, so reasoning is judged implicitly during training, with no process labels and no reward model. Because both models are optimized, each faces a progressively stronger rival, which single-model RL cannot provide. The two need only be comparably strong and behaviorally different. At inference the pair deploys as it trains, a two-stage cascade in which one model drafts and the other answers after reading the draft. On the hard split of DeepMath with Qwen3, this doubles GRPO's pass@1, roughly eight times the gain of an untrained Mixture-of-Agents pass over the same base. The ordering replicates on competitive-programming code and across model families (Qwen3.5, Gemma 4). For now the models talk in text; the next step is to let them reason together in latent space.

强化学习推理评估双模型竞争自主评测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。