用多样性策略优化LLM应用测试，减少成本并提升覆盖率。
Adaptive Testing for LLM-Based Applications: A Diversity-based Approach
- 基于字符串距离的自适应测试，动态选择新测试用例。
- 在相同测试预算下，发现更多缺陷，输出更多样化。
- 适合需要高效测试的LLM应用开发者和质量保障团队。
大型语言模型（LLMs）驱动的软件系统兴起，催生了多种测试框架，主要将提示模板作为测试单元。尽管测试输入执行与输出评估成本高昂，现有工具仍忽视测试用例集的优化，亟需定制化的测试选择或优先级策略。本文表明，如自适应随机测试（ART）等基于多样性的测试技术，结合合适的字符串距离度量，可有效应用于提示模板测试。我们提出的自适应测试方法通过分析已有测试集及其标注结果，生成评分以指导新测试用例的选择。在多种字符串距离实现下的实验结果证实，该方法可在更小的测试预算下发现更多故障，并促进生成更丰富的输出。
原文摘要 · Abstract (English)
The recent surge of building software systems powered by Large Language Models (LLMs) has led to the development of various testing frameworks, primarily focused on treating prompt templates as the unit of testing. Despite the significant costs associated with test input execution and output assessment, the curation of optimized test suites is yet overlooked in these tools, which calls for tailored test selection or prioritization strategies. In this paper, we show that diversity-based testing techniques, such as Adaptive Random Testing (ART) with appropriate string distance metrics, can be effectively applied to the testing of prompt templates. Our proposed adaptive testing approach adjusts the conventional ART process to this context by selecting new test inputs based on scores derived from existing test suite and their labelling results. Our results, obtained using various implementations that explore several string-based distances, confirm that our approach enables the discovery of failures with reduced testing budgets and promotes the generation of more varied outputs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。