用难度感知强化学习提升代码验证效率，小模型也能超大模型表现
CVeDRL: An Efficient Code Verifier via Difficulty-aware Reinforcement Learning
- 基于语法和功能的奖励设计，结合难度感知的指数奖励调节
- 0.6B参数模型实现28.97%更高通过率、15.08%更高分支覆盖率
- 推理速度超基线20倍，适合高效代码生成后验证场景
代码验证在基于大语言模型的代码生成后验证中至关重要，但现有监督微调方法存在数据稀缺、失败率高、推理效率低等问题。强化学习虽能通过执行驱动奖励实现无监督优化，但仅使用功能奖励的朴素方法难以生成复杂分支的有效单元测试。我们首次理论分析表明，分支覆盖率、样本难度、语法与功能正确性可共同建模为强化学习奖励信号，优化这些信号可提升基于单元测试验证的可靠性。据此，我们设计了语法与功能感知奖励，并提出基于分支和样本难度感知的强化学习方法，采用指数奖励塑造和静态分析指标。该方法仅用0.6B参数即达到业界领先性能，在通过率上比GPT-3.5高出28.97%，分支覆盖率提升15.08%，推理速度超过同类基线20倍以上。代码已开源于https://github.com/LIGHTCHASER1/CVeDRL.git。
原文摘要 · Abstract (English)
Code verifiers play a critical role in post-verification for LLM-based code generation, yet existing supervised fine-tuning methods suffer from data scarcity, high failure rates, and poor inference efficiency. While reinforcement learning (RL) offers a promising alternative by optimizing models through execution-driven rewards without labeled supervision, our preliminary results show that naive RL with only functionality rewards fails to generate effective unit tests for difficult branches and samples. We first theoretically analyze showing that branch coverage, sample difficulty, syntactic and functional correctness can be jointly modeled as RL rewards, where optimizing these signals can improve the reliability of unit-test-based verification. Guided by this analysis, we design syntax- and functionality-aware rewards and further propose branch- and sample-difficulty--aware RL using exponential reward shaping and static analysis metrics. With this formulation, CVeDRL achieves state-of-the-art performance with only 0.6B parameters, yielding up to 28.97% higher pass rate and 15.08% higher branch coverage than GPT-3.5, while delivering over $20\times$ faster inference than competitive baselines. Code is available at https://github.com/LIGHTCHASER1/CVeDRL.git
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。