用提示词覆盖率提升大模型编程的测试效果,发现能多查30%以上缺陷。
Prompt Coverage Adequacy

- 通过分析大模型注意力机制,衡量测试用例对提示词需求的覆盖程度。
- 实验显示该方法比传统代码覆盖率多发现30%以上的软件缺陷。
- 适合研究或实践大模型驱动开发的测试人员使用。
近年来,大型语言模型(LLMs)和自主代理正推动软件开发向更高抽象层次演进,从编写精确代码转向表达意图与目标。这一范式转变带来了新挑战,尤其是在以提示词而非代码为主开发产物时,如何指导测试。为此,我们提出提示词覆盖率充分性(Prompt Coverage Adequacy),一种针对由任务描述生成代码的测试覆盖标准。该准则类比于传统代码覆盖率,但作用于大模型与代理编程中的提示词层面,通过利用大模型的注意力机制,衡量测试套件对提示词中需求的满足程度。我们在两个数据集和多个LLMs上评估了该准则的一个简单实现——基于注意力增强的方法。结果表明,提示词覆盖率与故障检测有效性相关,在引导测试生成时,可比传统代码覆盖率多发现30%以上的缺陷。这些发现表明,提示词覆盖率充分性可成为适配大模型驱动软件开发新范式的测试度量基础,克服经典覆盖率准则在此情境下的局限。
原文摘要 · Abstract (English)
In recent years, it has become increasingly evident that large language models (LLMs) and autonomous agents raise the level of abstraction in software development by shifting the focus from writing precise procedures to expressing intents and goals. This paradigm shift introduces new challenges, particularly in how testing should be guided when prompts, rather than code, become primary development artifacts. To address this challenge, we propose Prompt Coverage Adequacy, a novel coverage criterion designed to support the testing of code generated from task descriptions. Prompt Coverage Adequacy serves as an analog to traditional code coverage, but operates at the level of prompts used in LLM and agent-based programming. Specifically, it measures how well a given test suite satisfies the requirements expressed in a prompt by leveraging the attention mechanisms of LLMs. We evaluate a simple instantiation of this criterion, based on attention boosting, across two datasets and multiple LLMs. Our results demonstrate that Prompt Coverage is associated with fault-detection effectiveness and can uncover over 30+% more faults than traditional code coverage when used to guide test generation. These findings suggest that Prompt Coverage Adequacy can serve as a foundation for developing testing metrics better suited to the emerging paradigm of LLM-driven software development, addressing the limitations of classical coverage criteria in this new context.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。