用大模型生成能发现程序缺陷的测试判断标准，提升自动化测试质量。
Understanding LLM-Driven Test Oracle Generation
- 通过自然语言提示让大模型理解预期行为，生成测试判断依据。
- 不同提示策略和上下文信息显著影响生成判断的准确率。
- 适合关注大模型辅助测试、软件工程自动化的研究者阅读。
自动化单元测试旨在提升软件质量并减少手动编写测试的时间与精力。然而，现有技术主要生成基于被测类实现行为的回归断言，未能解决断言问题：即区分正确与错误程序行为的挑战。随着基础模型（FMs），尤其是大语言模型（LLMs）的兴起，生成反映预期行为的测试断言成为可能。这使大模型成为推动Promptware的关键，即通过自然语言提示驱动软件开发与测试。本文对大模型在生成可暴露软件缺陷的测试断言方面的有效性进行了实证研究，考察了不同提示策略及上下文输入水平对生成断言质量的影响。研究结果揭示了大模型在基础模型时代生成断言的能力与局限，深化了对其性能的理解，并为该领域的未来研究提供了支持。
原文摘要 · Abstract (English)
Automated unit test generation aims to improve software quality while reducing the time and effort required for creating tests manually. However, existing techniques primarily generate regression oracles that predicate on the implemented behavior of the class under test. They do not address the oracle problem: the challenge of distinguishing correct from incorrect program behavior. With the rise of Foundation Models (FMs), particularly Large Language Models (LLMs), there is a new opportunity to generate test oracles that reflect intended behavior. This positions LLMs as enablers of Promptware, where software creation and testing are driven by natural-language prompts. This paper presents an empirical study on the effectiveness of LLMs in generating test oracles that expose software failures. We investigate how different prompting strategies and levels of contextual input impact the quality of LLM-generated oracles. Our findings offer insights into the strengths and limitations of LLM-based oracle generation in the FM era, improving our understanding of their capabilities and fostering future research in this area.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。