arXiv:2508.20737cs.SEcs.AI2025-08被引 5

为大模型应用测试设计轻量级通信协议,解决多层架构下的质量保障难题。

Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocol

  • 分三层架构解析大模型应用,定位测试难点
  • 提出四类协作策略与闭环质量框架,提升测试有效性
  • 设计可嵌入现有框架的测试导向通信协议AICL

大语言模型应用已从简单文本生成演变为集成检索增强、工具调用和多轮交互的复杂系统。其固有的非确定性、动态性和上下文依赖性给质量保障带来根本挑战。本文将大模型应用分解为三层次架构:系统外壳层、提示编排层和大模型推理核心层。分析发现传统软件测试方法在各层适用性不同:外壳层直接可用,编排层需语义重构,推理核心层则需范式转变。对比软件工程与人工智能安全分析方法,揭示测试单元抽象、评估指标和生命周期管理的结构性断层。识别出四大本质差异,衍生出六大核心挑战。为此提出四类协同策略(保留、翻译、整合、运行时),并探索融合预部署验证与运行时监控的可信闭环质量保障框架。基于此,提供实践指导与协议建议,推动大模型应用测试标准化。提出用于智能体间通信的测试导向协议——代理交互通信语言(AICL),具备良好可集成性。

原文摘要 · Abstract (English)

Applications of Large Language Models~(LLMs) have evolved from simple text generators into complex software systems that integrate retrieval augmentation, tool invocation, and multi-turn interactions. Their inherent non-determinism, dynamism, and context dependence pose fundamental challenges for quality assurance. This paper decomposes LLM applications into a three-layer architecture: \textbf{\textit{System Shell Layer}}, \textbf{\textit{Prompt Orchestration Layer}}, and \textbf{\textit{LLM Inference Core}}. We then assess the applicability of traditional software testing methods in each layer: directly applicable at the shell layer, requiring semantic reinterpretation at the orchestration layer, and necessitating paradigm shifts at the inference core. A comparative analysis of Testing AI methods from the software engineering community and safety analysis techniques from the AI community reveals structural disconnects in testing unit abstraction, evaluation metrics, and lifecycle management. We identify four fundamental differences that underlie 6 core challenges. To address these, we propose four types of collaborative strategies (\emph{Retain}, \emph{Translate}, \emph{Integrate}, and \emph{Runtime}) and explore a closed-loop, trustworthy quality assurance framework that combines pre-deployment validation with runtime monitoring. Based on these strategies, we offer practical guidance and a protocol proposal to support the standardization and tooling of LLM application testing. We propose a protocol \textbf{\textit{Agent Interaction Communication Language}} (AICL) that is used to communicate between AI agents. AICL has the test-oriented features and is easily integrated in the current agent framework.

大模型测试智能体质量保障协议设计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。