arXiv:2605.11299cs.LGcs.CL2026-05

让模型通过比较自身生成结果来自我学习,提升代码生成与判断能力。

Primal Generation, Dual Judgment: Self-Training from Test-Time Scaling

论文配图:Primal Generation, Dual Judgment: Self-Training from Test-Time Scaling
图 1 · 摘自论文原文
  • 利用模型自生成的多个代码方案,通过对比执行结果进行训练。
  • 在LiveCodeBench上,单次生成准确率提升3.1,Best-of-4准确率提升4.1。
  • 适合想提升代码生成质量且不依赖人工标注的开发者或研究者。

代码生成通常在程序的原始空间中进行:模型生成候选解并接收稀疏的执行反馈(如仅通过/失败)。测试时缩放通过采样多个候选解并进行比较来增强推理,但这种比较信息在推理后被丢弃。我们提出,这些信息定义了一个对偶判断空间,能提供更丰富的训练信号——模型学习的不是孤立的成功或失败,而是自身多个合理尝试之间的相对正确性结构。为此,我们提出DuST(对偶自训练)框架:从模型自身分布中采样候选程序,通过沙箱执行打标签,保留包含成功与失败的组合,并使用GRPO训练模型按执行正确性排序。目标完全为判别式:模型从未被直接奖励生成正确程序。对偶自训练同时提升了判断与生成能力。在覆盖两个系列、三个规模(4B至30B)的五种模型上,DuST持续提升LiveCodeBench上的Best-of-4测试时缩放效果。对于Qwen3-30B-Thinking在LiveCodeBench v6上,判断质量提升+6.2 NDCG,单样本通过率提升+3.1,Best-of-4准确率提升+4.1。训练后单次运行表现达到原模型Best-of-4水平。在相同排名数据上进行SFT可提升判断但不改善生成,证实策略内强化学习是将对偶空间学习转化为原始生成的关键机制。

原文摘要 · Abstract (English)

Code generation is typically trained in the primal space of programs: a model produces a candidate solution and receives sparse execution feedback, often a single pass/fail bit. Test-time scaling enriches the inference procedure by sampling multiple candidates and judging among them, but the comparative information this process reveals is discarded after inference. We argue that this information defines a dual judgment space that provides a far richer training signal: the model learns not from an isolated success or failure, but from the relative correctness structure across its own plausible attempts, identifying which succeed, which fail, and what distinguishes them. We introduce DuST (Dual Self-Training), a framework for self-training from the dual judgment space. DuST samples candidate programs from the model's own distribution, labels them through sandbox execution, retains groups containing both successes and failures, and trains the model to rank candidates by execution correctness using GRPO. The objective is purely discriminative: the model is never directly rewarded for generating correct programs. Dual self-training improves both judgment and generation. Across five models spanning two families and three scales (4B to 30B), DuST consistently improves Best-of-4 test-time scaling on LiveCodeBench. For Qwen3-30B-Thinking on LiveCodeBench v6, judgment quality improves by +6.2 NDCG, single-sample pass@1 improves by +3.1, and Best-of-4 accuracy improves by +4.1. The trained model's single rollout matches the base model's Best-of-4 performance. SFT on the same ranking data improves judgment without improving generation, confirming that on-policy RL is the mechanism that transfers dual-space learning back into primal generation.

代码生成自训练测试时缩放强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。