arXiv:2608.20349cs.CLcs.AI2026-08ACL

发现提示词微调会剧烈影响大模型表现,提出提升稳定性的新方法。

Beyond Prompt Engineering: A Systematic Analysis of Prompt Lexical Sensitivity and Its Impacts on Quality

论文配图:Beyond Prompt Engineering: A Systematic Analysis of Prompt Lexical Sensitivity and Its Impacts on Quality
图 1 · 摘自论文原文
  • 通过13.2万组提示变体,分析词汇变化对模型影响的机制
  • 发现高平均性能对应低波动,提升稳定性可降低40.7%方差
  • 提出自动优化提示的代理工具,适合需稳定输出的应用场景

大语言模型对提示词表面变化极为敏感,细微词汇调整可能引发性能大幅波动。我们首次开展大规模、基于n-gram词元级别的提示稳定性机制分析,使用包含13.2万组提示变体的数据集。研究揭示提示性能稳定性的基本规律:平均任务性能越高,其在提示扰动下的方差越小、鲁棒性越强。识别出两个关键语言驱动因素:(1) 领域术语,能严格锚定语义边界;(2) 明确的动作指令,可规范推理路径。二者共同缩小模型解释空间,实现更确定的生成行为。基于此,我们提出自动化提示重构代理,通过注入领域锚定与操作约束优化输入。实证评估显示,在代码生成任务中,该方法使性能方差降低40.7%,同时保持或提升平均性能。研究成果为实现可解释、数据支持的鲁棒提示工程提供了框架。

原文摘要 · Abstract (English)

Large Language Models (LLMs) exhibit extreme sensitivity to surface-level prompt variations, in which minor lexical changes can trigger disproportionate performance fluctuations. Moving beyond black-box optimization and coarse-grained templates, we present the first large-scale, n-gram token-level mechanistic analysis of prompt stability, leveraging a dataset of 132,000 prompt variants. Our investigation reveals a fundamental Scaling Law of Prompt Performance Stability: higher average task performance is strongly associated with lower variance and greater robustness across prompt perturbation. We identify two core linguistic drivers underlying this robustness: (1) Domain-Specific Terminology, which tightly anchors semantic boundaries, and (2) Explicit Action Directives, which formalize reasoning trajectories. Together, these elements constrain the model's interpretative space, effectively ``locking in'' more deterministic generation behavior. Building on these insights, we introduce an automated Prompt-Refining Agent that systematically restructures input queries by injecting domain anchoring and operational constraints. Empirical evaluation shows that our approach reduces performance variance by 40.7% in code generation task, while preserving or improving mean performance. These findings provide a statistically grounded and mechanistically interpretable framework for achieving robust prompt engineering.

提示工程大模型稳定性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。