用验证反馈强化学习，让小模型生成更正确的代码。
Improving Small Language Models for Code Generation with Reinforcement Learning from Verification Feedback
- 通过测试结果和代码检查结合的奖励机制训练模型。
- 在MBPP上使代码通过率提升13个百分点。
- 发现代码风格优化可能影响正确性，需综合评估。
基于可验证奖励的强化学习（RLVR）利用程序可检查的信号（如单元测试结果）直接优化代码生成的功能正确性。我们在MBPP基准上，对两个小型模型（Qwen3-0.6B 和 Llama3.2-1B）采用LoRA微调，考察了多种奖励设计：仅使用单元测试奖励、仅使用Ruff代码检查器进行静态分析引导，以及两者结合的奖励。对比了分组策略优化方法（GRPO与GSPO），并评估功能正确性与行为诊断指标。实验表明，在所提出的组合奖励配置下，MBPP测试集上的pass@1最高提升13个百分点。然而，仅使用静态分析惩罚会引发系统性偏差：模型倾向于生成更短的代码以减少代码警告，但未必提升功能性正确性。而组合奖励能缓解此退化现象，实现正确性与代码风格约束间的更稳定权衡。结果表明，RLVR在代码生成中的效果高度依赖奖励设计与优化粒度，且除pass@1外，生成长度、Ruff严重性分布及执行错误类型等诊断指标对识别失败模式具有重要价值。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) trains language models using programmatically checkable signals such as unit-test outcomes, enabling direct optimization for functional correctness in code generation. We conduct an empirical study of RLVR for Python code generation on the MBPP benchmark using two small models (Qwen3-0.6B and Llama3.2-1B) with LoRA fine-tuning. Across multiple reward formulations such as: unit-test-only rewards, static-analysis-only shaping via the Ruff linter, and a combined reward, we compare group-based policy optimization variants (GRPO and GSPO) and evaluate both functional correctness and behavioral diagnostics. In our experimental setting, RLVR improves pass@1 on MBPP test by up to 13 percentage points under proposed combined reward configuration. However, we find that reward shaping can induce systematic behavioral shifts: using only static-analysis penalties may bias the policy toward shorter completions that reduce lint errors without reliably improving functional correctness. In contrast, combined rewards mitigate this degeneration and yield more stable trade-offs between correctness and style constraints. Overall, our results highlight that RLVR effectiveness for code generation is highly sensitive to reward design and optimization granularity, and that diagnostics beyond pass@1, including generation length, Ruff severity profiles, and execution error types are useful for identifying failure modes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。