用奖励模型提升代码大模型训练与推理效率,不依赖测试用例也能大幅提效。
CodeScaler: Scaling Code LLM Training and Test-Time Inference via Reward Models
- 基于可验证反馈构建奖励模型,自动提取有效代码并优化奖励函数。
- 在4个基准上相比执行反馈强化学习提升1.55至14.64分,无需测试用例。
- 推理时延迟降为1/10,且在代码、通用和推理任务均超越现有模型。
基于可验证奖励的强化学习(RLVR)通过单元测试的执行反馈推动了代码大模型的发展,但其扩展性受限于高质量测试用例的可用性和可靠性。我们提出CodeScaler,一种用于扩展代码生成的强化学习训练与推理阶段的奖励模型。CodeScaler基于精心筛选的偏好数据训练,包含语法感知的代码提取和保持有效性的奖励塑造机制,确保优化过程稳定可靠。在四个编码基准测试中,CodeScaler在Qwen3-8B-Base上比执行反馈强化学习高出1.55分,在Qwen3-14B-Base上高出4.23分。通过进一步扩展至44,000个问题并引入合成数据,其性能较基础模型提升14.64分,且无需任何测试用例。在推理阶段,CodeScaler作为测试时扩展方法,性能媲美单元测试方案,同时实现10倍的延迟降低。此外,CodeScaler在RM-Bench评测中不仅在代码领域领先3.3分,通用和推理领域平均也领先2.7分。
原文摘要 · Abstract (English)
Reinforcement Learning from Verifiable Rewards (RLVR) has driven recent progress in code large language models by leveraging execution-based feedback from unit tests, but its scalability is fundamentally constrained by the availability and reliability of high-quality test cases. We propose CodeScaler, a reward model designed to scale both reinforcement learning training and test-time inference for code generation. CodeScaler is trained on carefully curated preference data derived from verified code problems and incorporates syntax-aware code extraction and validity-preserving reward shaping to ensure stable and robust optimization. Across four coding benchmarks, CodeScaler consistently outperforms execution-based RL by +1.55 points on Qwen3-8B-Base and +4.23 points on Qwen3-14B-Base. By further scaling to 44K problems with additional synthetic data, CodeScaler yields +14.64 points improvement over the base model without requiring any test cases. At inference time, CodeScaler serves as an effective test-time scaling method, achieving performance comparable to unit test approaches while providing a 10-fold reduction in latency. Moreover, CodeScaler surpasses existing reward models on RM-Bench not only in the code domain (+3.3 points), but also in general and reasoning domains (+2.7 points on average).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。