通过回归分析量化提示词各部分对大模型表现的影响。
A Regression Framework for Understanding Prompt Component Impact on LLM Performance
- 用回归模型分析提示词不同部分对模型性能的影响。
- 两个模型的提示词部分可解释72%~77%的性能差异。
- 错误示例会显著干扰模型，正负指令影响相反。
随着大语言模型（LLMs）持续改进并深入集成到软件系统中，理解其在何种条件下表现良好变得愈发重要。本文提出一种统计框架，用于分析特定提示特征对LLM性能的影响。该方法扩展了可解释人工智能（XAI）技术，通过拟合回归模型，将提示词的不同部分与模型评估结果关联。我们以Mistral-7B和GPT-OSS-20B两个开源模型为例，研究其解决简单算术问题时如何利用提示。个体提示部分的回归模型分别解释了72%和77%的模型性能变化。研究发现，包含错误示例的提示会阻碍模型完成任务；而正向与负向指令对性能的影响并不显著差异，且存在相互矛盾的作用。该框架可为关键场景中的决策者提供细粒度洞察，了解提示如何影响模型完成任务。
原文摘要 · Abstract (English)
As large language models (LLMs) continue to improve and see further integration into software systems, so does the need to understand the conditions in which they will perform. We contribute a statistical framework for understanding the impact of specific prompt features on LLM performance. The approach extends previous explainable artificial intelligence (XAI) methods specifically to inspect LLMs by fitting regression models relating portions of the prompt to LLM evaluation. We apply our method to compare how two open-source models, Mistral-7B and GPT-OSS-20B, leverage the prompt to perform a simple arithmetic problem. Regression models of individual prompt portions explain 72% and 77% of variation in model performances, respectively. We find misinformation in the form of incorrect example query-answer pairs impedes both models from solving the arithmetic query, though positive examples do not find significant variability in the impact of positive and negative instructions - these prompts have contradictory effects on model performance. The framework serves as a tool for decision makers in critical scenarios to gain granular insight into how the prompt influences an LLM to solve a task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。