arXiv:2503.00481cs.SEcs.AI2025-03被引 19

为大模型软件测试构建四维分类体系,揭示现有工具的三大缺陷

Challenges in Testing Large Language Model Based Software: A Faceted Taxonomy

  • 提出输入输出双重模糊性下的测试用例四维度分类法
  • 发现当前工具忽略模型版本与运行变异,仅做孤立测试
  • 适合研究大模型测试的开发者和学术界同行

大语言模型(LLMs)和多智能体大模型(MALLMs)引入了传统软件或机器学习系统中不存在的非确定性,仅靠输出对比或测试集统计准确率已无法验证正确性。本文基于文献与实践经验,构建了大模型测试用例设计的分类体系。通过分析六个开源测试框架、对基于代理的系统进行配置敏感性研究,并对比原子与聚合测试用例的实例,识别出影响测试正确性的关键变异点。分类体系包含四个维度:目标、被测系统、输入及两种核心评估机制——原子型与聚合型。结果表明,现有工具将测试执行视为孤立事件,缺乏显式的聚合机制,且未能充分捕捉模型版本、配置及重复运行中的变异性。这凸显了需将正确性理解为结果分布而非二元判断,亟需学界与产业界合作建立具备变异感知能力的成熟测试方法。

原文摘要 · Abstract (English)

Large Language Models (LLMs) and Multi-Agent LLMs (MALLMs) introduce non-determinism unlike traditional or machine learning software, requiring new approaches to verifying correctness beyond simple output comparisons or statistical accuracy over test datasets. This paper presents a taxonomy for LLM test case design, informed by research literature and our experience. Each facet is exemplified, and we conduct an LLM-assisted analysis of six open-source testing frameworks, perform a sensitivity study of an agent-based system across different model configurations, and provide working examples contrasting atomic and aggregated test cases. We identify key variation points that impact test correctness and highlight open challenges that the research, industry, and open-source communities must address as LLMs become integral to software systems. Our taxonomy defines four facets of LLM test case design, addressing ambiguity in both inputs and outputs while establishing best practices. It distinguishes variability in goals, the system under test, and inputs, and introduces two key oracle types: atomic and aggregated. Our findings reveal that current tools treat test executions as isolated events, lack explicit aggregation mechanisms, and inadequately capture variability across model versions, configurations, and repeated runs. This highlights the need for viewing correctness as a distribution of outcomes rather than a binary property, requiring closer collaboration between academia and practitioners to establish mature, variability-aware testing methodologies.

大模型测试非确定性测试分类

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。