用大模型自动生成高质量测试用例，提升代码验证效果。
HardTests: Synthesizing High-Quality Test Cases for LLM Coding
- 用大模型构建测试生成流水线，自动合成高质测试用例。
- 生成测试在精度上比现有方法高11.3个百分点，召回率高17.5个百分点。
- 特别适合训练和评估复杂编程问题的代码生成模型。
验证器在大语言模型（LLM）推理中起关键作用，尤其在后训练技术如强化学习中不可或缺。然而，对于复杂编程题，可靠的验证器难以获得，因为精心伪装的错误解法仅能通过人工设计的边缘案例发现，而这些案例难以自动生成。为此，我们提出HARDTESTGEN，一种利用大模型进行高质量测试合成的流水线。基于该流程，我们构建了包含47,000道题目的综合性竞赛编程数据集HARDTESTS，其测试用例为合成的高质量测试。与现有测试相比，使用HARDTESTGEN生成的测试在评估LLM生成代码时，精度提升11.3个百分点，召回率提升17.5个百分点；在更难的问题上，精度提升可达40个百分点。HARDTESTS在下游代码生成任务中也表现出更强的模型训练效果。相关数据集与合成流水线将开源至https://leililab.github.io/HardTests/。
原文摘要 · Abstract (English)
Verifiers play a crucial role in large language model (LLM) reasoning, needed by post-training techniques such as reinforcement learning. However, reliable verifiers are hard to get for difficult coding problems, because a well-disguised wrong solution may only be detected by carefully human-written edge cases that are difficult to synthesize. To address this issue, we propose HARDTESTGEN, a pipeline for high-quality test synthesis using LLMs. With this pipeline, we curate a comprehensive competitive programming dataset HARDTESTS with 47k problems and synthetic high-quality tests. Compared with existing tests, HARDTESTGEN tests demonstrate precision that is 11.3 percentage points higher and recall that is 17.5 percentage points higher when evaluating LLM-generated code. For harder problems, the improvement in precision can be as large as 40 points. HARDTESTS also proves to be more effective for model training, measured by downstream code generation performance. We will open-source our dataset and synthesis pipeline at https://leililab.github.io/HardTests/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。