arXiv:2511.20836cs.CLcs.AI2025-11被引 1

结构化提示能显著影响大模型评估结果,改变排行榜排名。

Structured Prompts Improve Evaluation of Language Models

  • 用多种结构化提示策略替代单一静态提示进行评估
  • 平均性能提升6%,5个基准测试排名发生改变
  • 链式思维提示贡献最大,高级优化器增益有限

随着大语言模型在各领域广泛应用,高质量的评估框架对部署决策至关重要。然而,现有框架如HELM通常采用单一静态提示,而模型表现高度依赖提示设计,导致评分可能反映提示选择而非模型真实能力。本文提出可复现的DSPy+HELM框架,通过五种提示方法,在七个基准上评估四款前沿模型和两款开源模型,对比现有HELM基线。结果显示,提示选择会显著影响评估结果:结构化提示使性能平均提升6%,5个基准的排行榜排名发生变化,主要收益来自引入链式思维提示,高级优化器带来的额外增益有限。这是首个系统性将结构化提示融入成熟评估框架并量化提示影响的研究。代码已开源:(i) DSPy+HELM评估工具(https://github.com/stanford-crfm/helm/pull/3893),(ii) 提示优化流水线(https://github.com/StanfordMIMI/dspy-helm)。

原文摘要 · Abstract (English)

As language models (LMs) are increasingly adopted across domains, high-quality benchmarking frameworks are essential for guiding deployment decisions. In practice, however, frameworks such as Holistic Evaluation of Language Models (HELM) typically evaluate models under a single static prompt configuration, even though model behavior depends strongly on prompt choice. As a result, reported scores can reflect prompt choice as much as model capability. Declarative prompting frameworks such as DSPy offer a scalable way to evaluate models under a set of structured prompting strategies rather than a static prompt configuration. We present a reproducible DSPy+HELM framework for studying how prompt choice impacts reported benchmark outcomes. Using five prompting methods, we evaluate four frontier and two open-source LMs across seven benchmarks against existing HELM baseline scores. By evaluating LMs across a family of prompt configurations, we find that prompt choice can materially impact leaderboard outcomes. In particular, structured prompting improves performance (by 6% on average), alters comparisons (leaderboard rankings shift on 5/7 benchmarks), with most gains coming from introducing chain-of-thought, and little additional benefit from more advanced optimizers. To our knowledge, this is the first study to systematically integrate structured prompting into an established evaluation framework and quantify how prompt choice alone can impact benchmark conclusions. We open-source (i) DSPy+HELM Evaluation (https://github.com/stanford-crfm/helm/pull/3893) and (ii) Prompt Optimization Pipeline (https://github.com/StanfordMIMI/dspy-helm).

大模型评估提示工程排行榜可靠性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。