用强化学习优化代码性能,突破执行时间测量难题。
Reinforcement Learning for Code Optimization

- 设计可学习的测试环境与离线模拟器,解决时延奖励噪声问题
- 在Qwen 7B和CWM 32B上使通过率提升至31.3%和50.4%
- 兼顾正确性,性能优于标准RL方法,接近人类一半优化水平
代码正确性强化学习已成熟:模型生成程序,通过隐藏测试用例后给予通过奖励。将其扩展至代码优化看似直接——只需在奖励中加入执行时间。但实践中,一旦以时长为驱动,测量噪声、奖励稀疏或GRPO不稳定性会淹没信号,导致生成代码几乎无提速,且失败率上升。本文通过三阶段实现可学习的执行时间建模:(1)构建包含大规模优化测试的DMC-Optim与校准沙盒,改进测试方式;(2)在强化学习环境中融合正确性与速度奖励,并使用离线模拟器预测最优配置;(3)针对更稀疏、更嘈杂的时延执行场景,调整GRPO与评估机制。在DMC-Optim上,最强优化配置使Qwen 2.5 7B的严格top-50% pass@1从18.0%提升至31.3%,CWM 32B从30.7%升至50.4%;在更严苛的top-30%下,CWM 32B相对提升达125%。当沙盒退化时,鲁棒优化强化学习相比标准RLVR提升100%~200%。在LCB数据集上,CWM 32B在中位样本速度对比中胜出83%。相对于每题最快的人类正确提交,其复杂度改进速度达到人类的一半(14% vs 28%)。
原文摘要 · Abstract (English)
RL for code correctness is now established: have the model generate a program, run it against hidden test cases, and reward solutions that pass. Extending this to code optimization seems straightforward: just add execution time to the reward. But in practice, once timing drives the reward, small problems in measurement noise, reward sparsity, or GRPO instability overwhelm the signal and make RL fail: generated solutions are barely faster, and more of them can fail. We make execution time learnable through three stages: (1) how code is tested, by building DMC-Optim with large optimization tests and a calibrated sandbox; (2) how speed is turned into reward, by composing correctness and speed in the RL environment and using an offline simulator to predict the most promising configurations; and (3) how the model learns from that reward, by adapting GRPO and evaluation to the sparser, noisier timed-execution setting. On DMC-Optim, the strongest optimization-aware configurations improve strict top-50% pass@1 from 18.0% to 31.3% on Qwen 2.5 7B and from 30.7% to 50.4% on CWM 32B. These gains further increase at stricter percentiles such as top-30%, with 125% relative improvement for CWM 32B, while preserving pure-correctness scores. When the timing sandbox is degraded, robust optimization RL reaches 100% to 200% improvement over standard RLVR, depending on the evaluation criterion. On LCB, CWM 32B wins up to 83% of median-sample speed comparisons against standard RLVR. Relative to the fastest correct human submissions per problem, it reaches about half the human rate of complexity-class improvements (14% vs. 28%).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。