测试大模型能否生成有效代码检测用例，发现其针对漏洞的生成能力远不如人类。
Can LLMs Generate Reliable Test Case Generators? A Study on Competition-Level Programming Problems
- 构建竞赛级编程问题的用例生成基准，评估大模型生成有效用例的能力
- 顶尖大模型虽能生成合规用例，但对暴露人工代码缺陷的针对性用例生成效果差
- 提供高质量人工标注指令数据集，可显著提升大模型在目标用例生成上的表现
大型语言模型（LLMs）在代码生成方面表现出色，但在通过用例生成进行代码检查或调试方面的应用仍不明确。本文从竞赛级编程（CP）问题出发，提出TCGBench——一个用于评估LLM生成用例生成器能力的基准。该基准包含两项任务：一是为给定的CP问题生成有效的用例生成器；二是生成能暴露人工代码缺陷的目标用例生成器。实验结果表明，尽管先进模型多数情况下可生成有效用例，但它们在生成能够揭示人工代码缺陷的针对性用例方面表现不佳。即使使用如o3-mini等先进推理模型，其表现也显著低于人类水平。此外，本文构建了一个高质量、人工精校的针对性用例生成指令数据集。分析显示，该数据集通过提示工程和微调均可有效提升大模型在该任务上的性能。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, capable of tackling complex tasks during inference. However, the extent to which LLMs can be utilized for code checking or debugging through test case generation remains largely unexplored. We investigate this problem from the perspective of competition-level programming (CP) programs and propose TCGBench, a Benchmark for (LLM generation of) Test Case Generators. This benchmark comprises two tasks, aimed at studying the capabilities of LLMs in (1) generating valid test case generators for a given CP problem, and further (2) generating targeted test case generators that expose bugs in human-written code. Experimental results indicate that while state-of-the-art LLMs can generate valid test case generators in most cases, most LLMs struggle to generate targeted test cases that reveal flaws in human code effectively. Especially, even advanced reasoning models (e.g., o3-mini) fall significantly short of human performance in the task of generating targeted generators. Furthermore, we construct a high-quality, manually curated dataset of instructions for generating targeted generators. Analysis demonstrates that the performance of LLMs can be enhanced with the aid of this dataset, by both prompting and fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。