评测大模型时不用优化提示词，结果可能误导决策。
Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleading
- 评测时对每个模型单独优化提示词，而非用统一模板。
- 同一模型在不同提示词下排名差异可达数名。
- 适合做模型选型或部署的工程师参考。
当前大模型评测框架对所有模型使用相同的静态提示模板，这与工业界为提升应用性能而对每个模型进行提示词优化（Prompt Optimization, PO）的实践不符。本文研究了提示词优化对大模型评测的影响。在公开学术基准和内部行业基准上的实验表明，提示词优化显著改变模型最终排名。这一发现强调，在选择特定任务的最佳模型时，必须针对每个模型进行提示词优化，以获得准确评估结果。
原文摘要 · Abstract (English)
Current Large Language Model (LLM) evaluation frameworks utilize the same static prompt template across all models under evaluation. This differs from the common industry practice of using prompt optimization (PO) techniques to optimize the prompt for each model to maximize application performance. In this paper, we investigate the effect of PO towards LLM evaluations. Our results on public academic and internal industry benchmarks show that PO greatly affects the final ranking of models. This highlights the importance of practitioners performing PO per model when conducting evaluations to choose the best model for a given task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。