arXiv:2606.27369cs.LG2026-06

无需标准答案,用评分优化训练大模型代码能力。

Reinforcement Learning without Ground-Truth Solutions can Improve LLMs

论文配图:Reinforcement Learning without Ground-Truth Solutions can Improve LLMs
图 1 · 摘自论文原文
  • 用实例间对比和排名反馈替代标准答案进行强化学习。
  • 在12个算法竞赛任务上提升大模型评分排名8.9%~9.4%。
  • 适合想提升代码泛化能力但无正确解场景的研究者。

基于可验证奖励的强化学习(RLVR)通常依赖标准答案提供奖励,限制了其在无标准解任务中的应用。本文提出一种排名诱导可验证框架(RiVER),在无需标准答案的情况下,通过确定性执行反馈作为连续值监督,训练大模型解决基于评分的优化任务。针对群体相对强化学习中出现的规模主导(得分量级未校准)和频率主导(次优解重复采样压制优质解)问题,RiVER采用基于实例的比较与奖励校准,强调最优解并保留对其他有效解的有界反馈。在12个AtCoder启发式竞赛任务上训练,评估显示:在ALE-Bench、LiveCodeBench和USACO上,Qwen3-8B和GLM-Z1-9B-0414分别提升8.9%和9.4%的评分排名。更重要的是,尽管训练仅使用评分任务且无标准答案,模型在精确解基准测试上仍实现平均2.4%和3.5%的绝对提升。相比之下,使用原始执行得分的基线虽在评分任务上有进步,但无法迁移至精确解任务。结果表明,经过恰当奖励校准的评分优化任务,可成为无需标准答案训练通用编码能力的有效环境。

原文摘要 · Abstract (English)

Reinforcement learning with verifiable rewards (RLVR) for training LLMs typically rely on ground-truth answers to assign rewards, limiting their applicability to tasks where the ground-truth solution is unknown. We introduce a \textbf{R}anking-\textbf{i}nduced \textbf{VER}ifiable framework (RiVER) that trains LLMs on score-based optimization tasks without ground-truth solutions, using deterministic execution feedback as continuous-valued supervision. When applying group-relative RL to such continuous rewards, we identify two key challenges: \emph{scale dominance}, where uncalibrated score magnitudes across test instances distort policy updates, and \emph{frequency dominance}, where repeatedly sampled suboptimal solutions can outweigh rare but stronger candidates. RiVER addresses these challenges with calibrated reward shaping that uses instance-wise comparisons and emphasizes top-ranked solvers while retaining bounded feedback for other valid solutions. We train on 12 AtCoder Heuristic Contest tasks and evaluate on Algorithm Engineering Benchmark (ALE-Bench), LiveCodeBench, and USACO. RiVER advances Qwen3-8B and GLM-Z1-9B-0414 by 8.9\% and 9.4\% in ALE rating rank. More importantly, despite training exclusively on score-based tasks without any ground-truth solutions, RiVER also improves the backbones across exact-solution benchmarks such as LiveCodeBench and USACO by an absolute average improvement of 2.4\% and 3.5\%. By contrast, baselines trained with raw execution scores improve ALE rating but fail to transfer to exact-solution benchmarks. These results suggest that score-based optimization tasks, combined with proper reward calibration, can serve as effective training environments for general coding ability without ground-truth solutions.

强化学习代码生成评分优化无标准答案

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。