用矩阵秩定义最小测试集,揭示大模型生成测试用例的真实短板。
How Many Code and Test Cases Are Enough? Evaluating Test Cases Generation from a Binary-Matrix Perspective
- 将错误代码与测试结果构建成二值矩阵,以矩阵秩确定最少需覆盖的错误模式数。
- 在百万级编程提交中构建出紧凑、多样且抗评分膨胀的基准测试集TC-Bench。
- 发现当前最强生成模型仅能覆盖约60%错误,暴露诊断能力严重不足。
评估大型语言模型自动生成的测试用例是一项关键但具有挑战性的任务。现有基准常在大规模、无结构的错误代码集合上评估排除率,导致计算成本高且得分虚高。此外,这些方法无意中奖励检测常见、简单错误的生成器,却未能惩罚其对罕见但关键缺陷的识别失败。本文提出两个核心问题:(1) 覆盖全部错误空间所需的最少错误代码集是多少?(2) 区分这些错误所需的最少测试用例数量是多少?我们引入一种新框架,将基准构建形式化为在二值代码-测试矩阵中寻找最优诊断基,其中行代表错误代码,列代表测试结果。该矩阵的秩给出了独立错误模式的最小数量,并为完全故障覆盖所需测试用例数提供了紧致上界。目标是找到大小等于矩阵秩、内部多样性最大的基。针对此NP难问题,我们提出WrongSelect算法,高效近似选择最具多样性的错误代码。基于数百万个编程竞赛提交应用该框架,我们构建了TC-Bench——一个紧凑、多样、抗评分膨胀的基准。大量实验表明,即使最先进的测试用例生成方法在TC-Bench上的排除率也仅为约60%,暴露出其诊断能力的重大差距,凸显未来改进的巨大空间。数据集地址:https://huggingface.co/datasets/Luoberta/TC-Bench,代码地址:https://github.com/Luowaterbi/TC-Bench。
原文摘要 · Abstract (English)
Evaluating test cases automatically generated by Large Language Models (LLMs) is a critical yet challenging task. Existing benchmarks often evaluate the exclusion ratio on large, unstructured collections of wrong codes, suffering from high computational costs and score inflation. Furthermore, they inadvertently reward generators that detect common, trivial bugs, while failing to penalize their inability to identify rare yet critical faults. In this work, we connect two fundamental questions: (1) What is the minimal set of wrong codes sufficient to represent the entire error space? and (2) What is the minimal set of test cases needed to distinguish them? We introduce a novel framework that formalizes benchmark construction as finding an optimal diagnostic basis in a binary code-test matrix, where rows represent wrong codes and columns represent test case results. The rank of this matrix specifies the minimal number of independent error patterns (wrong codes) and provides a tight upper bound on the number of test cases required for complete fault coverage. Our objective is to identify a basis of size equal to the matrix rank that maximizes internal diversity. To tackle this NP-hard problem, we propose WrongSelect, an efficient approximation algorithm to select maximally diverse wrong codes. Applying this framework to millions of competitive programming submissions, we construct TC-Bench, a compact, diverse, and inflation-resistant benchmark. Extensive experiments show that even the most advanced test case generation methods achieve only ~60% exclusion rates on TC-Bench, exposing a significant gap in their diagnostic power and highlighting substantial room for future improvement. Our dataset is available at: https://huggingface.co/datasets/Luoberta/TC-Bench and our code is at: https://github.com/Luowaterbi/TC-Bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。