通过对抗演化提升代码与测试生成质量,避免自欺欺人式测试。
Code-A1: Adversarial Evolving of Code LLM and Test LLM via Reinforcement Learning
- 分离代码与测试模型,让它们目标相反地协同进化。
- 在Qwen2.5-Coder上表现超越人工测试训练模型,测试发现缺陷能力更强。
- 适合需要高可靠性代码生成与自动化测试的工程场景。
代码生成的强化学习依赖单元测试通过率作为可验证奖励,但高质量测试集稀缺,现有数据集覆盖有限,且静态奖励无法随模型进步而调整。近期自对弈方法将代码与测试生成整合于单一模型,却面临固有困境:白盒访问导致模型生成简单测试以获取奖励(自合谋),而黑盒限制则产生通用测试,难以捕捉实现特定漏洞。本文提出Code-A1,一种对抗共演化框架,联合优化代码大模型(Code LLM)与测试大模型(Test LLM),二者目标相反:代码模型因通过更多测试被奖励,测试模型因揭露更多缺陷被奖励。这种架构分离消除了自合谋风险,安全支持白盒测试生成——测试模型可检查候选代码并设计针对性对抗测试。我们还引入错误本机制用于经验回放,并采用复合奖励平衡测试有效性与对抗难度。在Qwen2.5-Coder模型上的实验表明,Code-A1在代码生成性能上达到或超过基于人工标注测试训练的模型,同时显著提升测试生成能力。
原文摘要 · Abstract (English)
Reinforcement learning for code generation relies on verifiable rewards from unit test pass rates. Yet high-quality test suites are scarce, existing datasets offer limited coverage, and static rewards fail to adapt as models improve. Recent self-play methods unify code and test generation in a single model, but face a inherent dilemma: white-box access leads to self-collusion where the model produces trivial tests for easy rewards, yet black-box restriction yields generic tests that miss implementation-specific bugs. We introduce Code-A1, an adversarial co-evolution framework that jointly optimizes a Code LLM and a Test LLM with opposing objectives. The Code LLM is rewarded for passing more tests, while the Test LLM is rewarded for exposing more defects. This architectural separation eliminates self-collusion risks and safely enables white-box test generation, where the Test LLM can inspect candidate code to craft targeted adversarial tests. We further introduce a Mistake Book mechanism for experience replay and a composite reward balancing test validity with adversarial difficulty. Experiments on Qwen2.5-Coder models demonstrate that Code-A1 achieves code generation performance matching or exceeding models trained on human-annotated tests, while significantly improving test generation capability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。