用对抗性代码生成测试集,让大模型代码能力暴露真实水平
VeriScale: Adversarial Test-Suite Scaling for Verifiable Code Generation

- 通过对抗性实现自动扩充测试用例,提升覆盖多样性
- 测试集扩展超83倍,模型在新基准上得分大幅下降
- 提供轻量版可低耗评估,适合研究与工程验证
随着大语言模型在软件工程中的广泛应用,构建高质量评估基准对检验生成代码的功能正确性和形式可验证性至关重要。然而,现有基准受限于正负例测试用例的数量与质量,导致模型能力被高估。为此,我们提出VeriScale框架,基于对抗性实现,包含两阶段:测试集扩展以生成多样且具有挑战性的测试用例,以及测试集压缩以提炼出紧凑而具区分力的测试套件。该框架具通用性,我们将其应用于Verina,构建了VerinaPlus(测试集扩展超83倍)和VerinaLite(轻量14倍变体)。在八种先进LLM上的实验表明,VerinaPlus揭示了原基准下隐藏的模型弱点,表现为SpecGen与CodeGen任务得分显著下降;而VerinaLite在极低评估成本下仍保持强区分能力。相关增强基准与源码已公开于https://github.com/XiaoyangLiu-sjtu/VeriScale。
原文摘要 · Abstract (English)
As large language models (LLMs) are increasingly deployed for software engineering, constructing high-quality benchmarks is crucial for evaluating not just the functional correctness, but also the formal verifiability of generated code. However, existing benchmarks are limited by the quantity and quality of positive and negative test cases, leading to an overestimation of model capabilities in generating specifications and implementations. To address this, we propose VeriScale, a novel framework driven by the adversarial implementations. It consists of two stages: test-suite expansion to construct diverse and challenging test cases, and test-suite reduction to distill them into compact yet discriminative suites. While VeriScale is general, we instantiate it on Verina to construct VerinaPlus, which expands the original test suites by over 83$\times$, and VerinaLite, a lightweight 14$\times$ variant. Our experiments across eight state-of-the-art LLMs demonstrate that VerinaPlus exposes substantial model weaknesses hidden by the original benchmark, evidenced by sharp score drops on both SpecGen and CodeGen tasks, whereas VerinaLite maintains this discriminative power at a fraction of the evaluation cost. The enhanced benchmarks and source code are publicly available at https://github.com/XiaoyangLiu-sjtu/VeriScale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。