用强化学习让大模型自动生成带验证逻辑的测试代码
HarnessLLM: Automatic Testing Harness Generation via Reinforcement Learning
- 分两阶段训练大模型，生成包含输入与输出验证的测试框架
- 相比传统方法，能发现更多缺陷且测试策略更丰富
- 适合提升代码生成模型的可靠性，尤其在推理阶段
现有基于大模型的自动测试生成方法主要生成输入-输出对，用于判断正确程序的行为。虽然简单，但生成的测试多样性有限，难以提供充分的调试信息。我们提出HarnessLLM，一种两阶段训练流程，使大模型能够编写测试框架代码。具体而言，大模型生成合成输入并验证输出的代码，支持复杂测试用例和灵活的输出校验（如不变量检查）。为此，我们采用SFT后接RLVR的训练方式，并设计定制化奖励函数。实验表明，HarnessLLM在缺陷发现能力与测试策略多样性上均优于传统的输入-输出测试方法。此外，利用生成的测试用例进行推理阶段验证，还能提升代码生成性能。代码已开源：https://github.com/UCSB-NLP-Chang/HarnessLLM.git。
原文摘要 · Abstract (English)
Existing LLM-based automatic test generation methods mainly produce input and expected output pairs to categorize the intended behavior of correct programs. Although straightforward, these methods have limited diversity in generated tests and cannot provide enough debugging information. We propose HarnessLLM, a two-stage training pipeline that enables LLMs to write harness code for testing. Particularly, LLMs generate code that synthesizes inputs and validates the observed outputs, allowing complex test cases and flexible output validation such as invariant checking. To achieve this, we train LLMs with SFT followed by RLVR with a customized reward design. Experiments show that HarnessLLM outperforms input-output-based testing in bug finding and testing strategy diversity. HarnessLLM further benefits the code generation performance through test-time scaling with our generated test cases as inference-phase validation. Our code is available at https://github.com/UCSB-NLP-Chang/HarnessLLM.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。