通用提示优化可能适得其反，需用评估驱动迭代。
When Generic Prompt Improvements Hurt: Evaluation-Driven Iteration for LLM Applications
- 构建最小可行评估套件，针对不同应用类型匹配故障模式与验证证据。
- 实测发现添加通用规则使Qwen模型在RAG任务中准确率从26/30降至9/30。
- 强调提示修改应视为潜在回归风险，部署前必须任务专属测试。
大型语言模型（LLM）应用的评估不同于传统软件测试，因其输出具有概率性、语义可变性，并对提示和模型变化敏感。本文提出最小可行评估套件（MVES），一种面向审计的应用级LLM评估框架。该框架将应用类别与故障模式、度量指标、所需产物及验证证据相联结，覆盖通用LLM应用、检索增强系统和智能体工作流。我们配套开发了可复现的本地评估工具包，涵盖结构化提取、RAG引用/内容合规性及指令遵循检查。使用Ollama搭载Llama 3 8B Instruct与Qwen 2.5 7B Instruct，在每套扩展至30个案例的五种提示条件下进行消融实验。结果显示：在测试环境下，通用提示增强不带来单调提升——更强的输出约束提示提升了两模型的严格提取表现，但部分通用规则条件下RAG引用与内容合规性下降；其中Qwen 2.5在用户提示中附加通用规则时，该指标从26/30降至9/30。研究支持评估驱动的提示迭代策略：提示变更应视为潜在回归风险，须经任务专属评估套件验证后方可部署。附带代码库包含测试套件、提示变体、评估工具、原始日志与复现脚本。
原文摘要 · Abstract (English)
Evaluating Large Language Model (LLM) applications differs from conventional software testing because outputs are probabilistic, semantically variable, and sensitive to prompt and model changes. This technical report proposes the Minimum Viable Evaluation Suite (MVES), an audit-oriented structure for application-level LLM evaluation. MVES links application categories to failure modes, metrics, required artifacts, and validation evidence across general LLM applications, retrieval-augmented systems, and agentic workflows. We pair the framework with a reproducible local evaluation harness covering structured extraction, RAG citation/content-compliance, and instruction-following checks. Using Ollama with Llama 3 8B Instruct and Qwen 2.5 7B Instruct, we evaluate five prompt conditions over expanded 30-case-per-suite ablations. The results show that, in the tested local conditions, generic prompt additions do not produce monotonic improvements: stronger output-contract prompts improve strict extraction for both models, while RAG citation/content-compliance declines under some generic-rule conditions. The largest observed decline occurs for Qwen 2.5 on RAG when generic rules are appended to the user prompt, from 26/30 to 9/30. These findings support evaluation-driven prompt iteration: prompt changes should be treated as potential regression risks and tested against task-specific suites before deployment. The accompanying repository contains the test suites, prompt variants, evaluation harness, raw result logs, and scripts needed to reproduce the reported local ablations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。