自动为大模型提示词生成测试用例，提升提示词可靠性
PromptPex: Automatic Test Generation for Language Model Prompts
- 基于大模型自动生成针对提示词的多样化测试用例
- 在四个不同模型上使无效输出率比基线高30%以上
- 帮助开发者理解提示词在不同模型下的行为差异
大语言模型（LLMs）广泛应用于各类应用中，其提示词以类似代码的形式嵌入软件系统。与传统软件不同，提示词输出依赖于具体模型，且修改后效果难以预测。为此，我们提出 PromptPex，一个基于 LLM 的工具，可自动为指定提示词生成并评估单元测试。PromptPex 从提示词中提取输入输出规范，生成多样、目标明确且有效的测试用例。这些测试能有效发现提示词变更导致的回归问题，并帮助理解不同模型对提示词的解释差异。我们在八个基准提示词上使用 PromptPex 生成测试，并评估其在四类不同模型上的表现：结果表明，生成的测试引发的无效输出比例显著高于精心设计的基线测试生成器。通过提取提示词的具体规范，PromptPex 使提示词编写者能清晰地验证和理解提示词的关键特性。源码已开源：https://github.com/microsoft/promptpex。
原文摘要 · Abstract (English)
Large language models (LLMs) are being used in many applications and prompts for these models are integrated into software applications as code-like artifacts. These prompts behave much like traditional software in that they take inputs, generate outputs, and perform some specific function. However, prompts differ from traditional code in many ways and require new approaches to ensure that they are robust. For example, unlike traditional software the output of a prompt depends on the AI model that interprets it. Also, while natural language prompts are easy to modify, the impact of updates is harder to predict. New approaches to testing, debugging, and modifying prompts with respect to the model running them are required. To address some of these issues, we developed PromptPex, an LLM-based tool to automatically generate and evaluate unit tests for a given prompt. PromptPex extracts input and output specifications from a prompt and uses them to generate diverse, targeted, and valid unit tests. These tests are instrumental in identifying regressions when a prompt is changed and also serve as a tool to understand how prompts are interpreted by different models. We use PromptPex to generate tests for eight benchmark prompts and evaluate the quality of the generated tests by seeing if they can cause each of four diverse models to produce invalid output. PromptPex consistently creates tests that result in more invalid model outputs than a carefully constructed baseline LLM-based test generator. Furthermore, by extracting concrete specifications from the input prompt, PromptPex allows prompt writers to clearly understand and test specific aspects of their prompts. The source code of PromptPex is available at https://github.com/microsoft/promptpex.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。