用两阶段强化学习自动生成能发现代码模型漏洞的测试用例。
Two-Stage Reinforcement Learning for Sound and Adversarial Test Generation in Code LLMs
- 分两阶段训练测试生成器：先生成正确解对应的测试，再针对模型缺陷生成反例。
- 在TACO和LiveCodeBench上提升10%以上通过率，优化推理时答案选择。
- 适合研究代码大模型鲁棒性、测试生成或对抗样本攻击的开发者使用。
强化学习（RL）通过可执行反馈显著提升了大语言模型（LLMs）的代码生成能力。现有反馈主要依赖特定测试用例，但高质量测试用例因需兼具正确性和区分性而稀缺。为此，我们探索利用学习到的模型自动构建测试用例。研究发现，该任务本质是对抗性强化学习问题：测试生成应基于当前求解器的失败模式，生成有效反例。为此提出两阶段强化学习框架Test Cases Scaling（TCS），两个阶段均从滚动策略对齐缓冲区中训练测试生成器：第一阶段生成与参考解一致的测试，第二阶段将缓冲区限制在当前失败模式下，学习生成反例测试。在TACO和LiveCodeBench数据集上，TCS显著提升pass@1指标及推理阶段的答案选择效果。此外，所学测试生成器还能有效筛选其他LLM输出结果。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has substantially advanced code generation with large language models (LLMs) through executable feedback. The feedback for coding problems mainly comes from specific test cases, where high-quality test cases are often scarce since they should be both sound and discriminative. We thus turn to study the auto-generation of test cases using the learned model. We find this is naturally an adversarial RL problem: the model is expected to generate effective test cases as counterexamples, depending on the solver's current failure modes. We propose Test Cases Scaling (TCS), a two-stage RL framework for effective test generation. Both stages train a test generator from a rolling policy-aligned buffer: Stage 1 generates tests consistent with the reference solution, and Stage 2 restricts the buffer to current failure modes and learns counterexample tests. Across TACO and LiveCodeBench, TCS improves both pass@1 and inference-time answer selection according to generated tests. We find the learned test generator also enables effective selection among other LLM outputs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。