arXiv:2605.28751cs.LGcs.AI2026-05

通过外推权重平均,拓展代码强化学习的正确性与效率边界。

Extrapolative Weight Averaging Reveals Correctness-Efficiency Frontiers in Code RL

  • 用嵌套单元测试训练不同覆盖度模型,生成可外推的性能前沿。
  • 外推后模型在难题上解决率提升3.3%,且跨多种推理模式有效。
  • 适合追求高效代码生成的开发者和自动化编程研究者。

线性插值已被证明能追踪多个目标间的帕累托前沿,但外推权重平均是否能在不进行额外强化学习训练的情况下,将该前沿延伸至推理时有用的新型检查点仍不清楚。我们在竞赛编程的强化学习中研究此问题,其中隐藏的单元测试在时间和内存限制下同时要求功能正确性和计算效率。从共同初始化出发,我们训练了在嵌套单元测试覆盖率下的检查点:低覆盖率奖励仅需通过小输入测试,高覆盖率奖励则需逐步通过更大输入直至完整测试集。这一过程揭示出正确性-效率前沿:在难题上,更高覆盖率奖励虽减少优化失败,却增加正确性失败,整体求解率基本不变。插值可恢复该前沿,而外推则将其延伸至训练终点之外。该前沿及其外推延续在三种推理设置(纯推理、工具使用、代理编码)和两种模型规模(32B与7B)下均成立。在问题层面,沿前沿移动会改变哪些问题被解决,使外推检查点成为推理时的互补策略。在匹配样本预算下,采用外推权重平均的集成模型在LCB/hard上的pass@250提升3.3%优于最佳单个检查点。结果表明,代码强化学习中的嵌套单元测试覆盖能诱导出可导航、可扩展并可利用的前沿。

原文摘要 · Abstract (English)

Linear interpolation between fine-tuned checkpoints has been shown to trace the Pareto front between competing objectives, but whether extrapolative weight averaging can extend such frontiers to new checkpoints useful at inference time, without additional RL training, remains unclear. We study this question in RL for competitive programming, where hidden unit tests under time and memory limits enforce both functional correctness and computational efficiency. Starting from a shared initialization, we train checkpoints under nested unit-test coverage: low-coverage rewards require passing smaller-input tests, while high-coverage rewards require passing progressively larger tests up to the full suite. This sweep reveals the emergence of a correctness-efficiency frontier: on hard problems, higher-coverage reward reduces optimization failures but increases correctness failures, leaving solve rate nearly unchanged. Interpolation between low- and high-coverage checkpoints recovers this frontier, while extrapolation extends it beyond the trained endpoints. Both the frontier and its extrapolative continuation appear across three inference settings, pure reasoning, tool use, and agentic coding, and across two model scales, 32B and 7B. At the problem level, moving along the frontier changes which problems are solved, making extrapolated checkpoints complementary policies in inference-time scaling. Ensembles with extrapolative weight averaging broaden coverage and improve pass@250 on LCB/hard by 3.3% over the best single checkpoint at matched sample budget. These results show that nested unit-test coverage in code RL induces a frontier that extrapolative weight averaging can navigate, extend, and exploit.

代码生成强化学习权重外推

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。