用语义熵自动检测大模型生成的无效测试用例，提升代码生成质量。
Toward Automated Validation of Language Model Synthesized Test Cases using Semantic Entropy
- 通过语义熵衡量测试用例的不确定性，判断其有效性。
- 在多个数据集上使测试有效率提升最高29%，代码通过率显著提高。
- 适合关注大模型生成测试可靠性的开发者和自动化测试研究者。
现代基于大语言模型（LLM）的编程代理常依赖测试执行反馈来优化生成的代码。这些测试用例由LLM合成生成，但可能产生无效或幻觉测试，误导反馈循环并降低代码优化效果。本文提出VALTEST框架，利用语义熵自动验证LLM生成的测试用例。通过分析测试用例的语义结构并计算基于熵的不确定性度量，训练机器学习模型分类测试用例为有效或无效，并过滤掉无效用例。在多个基准数据集和多种LLM上的实验表明，VALTEST不仅将测试有效率提升最高29%，还显著提高代码生成性能，体现在pass@1得分明显上升。大量实验还表明，语义熵是区分有效与无效测试用例的可靠指标，为提升软件测试和代码生成中LLM生成测试的正确性提供了稳健解决方案。
原文摘要 · Abstract (English)
Modern Large Language Model (LLM)-based programming agents often rely on test execution feedback to refine their generated code. These tests are synthetically generated by LLMs. However, LLMs may produce invalid or hallucinated test cases, which can mislead feedback loops and degrade the performance of agents in refining and improving code. This paper introduces VALTEST, a novel framework that leverages semantic entropy to automatically validate test cases generated by LLMs. Analyzing the semantic structure of test cases and computing entropy-based uncertainty measures, VALTEST trains a machine learning model to classify test cases as valid or invalid and filters out invalid test cases. Experiments on multiple benchmark datasets and various LLMs show that VALTEST not only boosts test validity by up to 29% but also improves code generation performance, as evidenced by significant increases in pass@1 scores. Our extensive experiments also reveal that semantic entropy is a reliable indicator to distinguish between valid and invalid test cases, which provides a robust solution for improving the correctness of LLM-generated test cases used in software testing and code generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。