让代码模型不仅写对，还学会写快，用性能反馈强化学习优化程序效率。
RLPF: Reinforcement Learning from Performance Feedback for Code Generation
- 设计分阶段奖励机制，失败代码按执行进度排序，正确代码按效率提升排名。
- 在PerfCodeBench上正确可运行率从11.1%提升至54.6%，效率提升从8.1%到38.6%。
- 适合关注代码生成效率与可优化性的研究者和工程实践者。
代码模型越来越多地使用执行反馈进行训练，但多数训练信号仍止于正确性。这留下一个重要缺口：两个程序可通过相同测试，却在运行时间上差异巨大。本文研究如何训练代码智能体优先选择更快的正确实现，而非仅将效率作为评估指标。主要难点在于运行时是脆弱的奖励信号——仅在程序正确后才有意义，任务间差异大，且多数未通过编译或运行的样本难以提供有效指导。为此提出RLPF（基于性能反馈的强化学习），将执行结果转化为分阶段奖励：失败程序按执行进度排序，正确程序则根据相对于基线到专家参考的相对改进程度排序。该机制在正确前提供有用反馈，正确后给予敏感性能反馈。在PerfCodeBench上对Qwen3-32B进行微调，使正确可运行解决方案从11.1%提升至54.6%，相对效率从8.1%提升至38.6%。训练后的模型达到与更强开源系统相当的竞争力，其优化行为在EffiBench-X上表现出适度迁移能力。额外实验表明，模型生成的参考虽有帮助但效果较弱，而复合奖励比仅考虑正确性或仅依赖运行时的基线更可靠。结果表明，代码智能体不仅能通过测试，还能主动优化所生成程序。
原文摘要 · Abstract (English)
Code models are increasingly trained with execution feedback, but most training signals still stop at correctness. This leaves an important gap for systems code: two programs can pass the same tests while differing greatly in runtime. We study how to train code agents to prefer faster correct implementations, rather than treating efficiency only as an evaluation metric. The key difficulty is that runtime is a fragile reward. It is meaningful only after a program is correct, varies across tasks, and gives little guidance when most sampled programs fail to compile or run. We propose \textbf{RLPF}, reinforcement learning from performance feedback, which turns execution outcomes into a staged reward. Failed programs are ordered by execution progress, while correct programs are ranked by their relative improvement from the baseline toward the expert reference. This gives useful feedback before correctness and performance-sensitive feedback after correctness. Fine-tuning Qwen3-32B with RLPF on PerfCodeBench raises correct-and-runnable solutions from $11.1\%$ to $54.6\%$ and improves relative efficiency from $8.1\%$ to $38.6\%$. The trained model becomes competitive with stronger open-weight systems, and its optimization behavior transfers modestly to EffiBench-X. Additional studies show that model-generated references provide useful but weaker supervision, and that the full composite reward is more reliable than correctness-only or runtime-only baselines. These results suggest that code agents can be trained not only to pass tests, but also to optimize the programs they write.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。