用统计方法提升大模型评估的可靠性，减少噪声干扰。
Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
- 将评估视为实验，引入统计学分析框架
- 提供模型对比差异的量化公式与实验设计建议
- 适合研究者在论文中规范报告评估结果
评估对理解大语言模型能力至关重要。本质上，评估是一种实验，但现有文献大多忽视了其他科学领域在实验分析与设计方面的研究成果。本文向具备一定统计学基础的研究者介绍如何思考和分析大语言模型评估数据。将评估问题视为来自未见总体的抽样，提出分析评估数据的公式，用于衡量两个模型之间的差异，并指导评估实验的设计。文章提出了若干具体建议，以最小化统计噪声、最大化评估结果的信息量，改进大模型评估的实施与报告方式。
原文摘要 · Abstract (English)
Evaluations are critical for understanding the capabilities of large language models (LLMs). Fundamentally, evaluations are experiments; but the literature on evaluations has largely ignored the literature from other sciences on experiment analysis and planning. This article shows researchers with some training in statistics how to think about and analyze data from language model evaluations. Conceptualizing evaluation questions as having been drawn from an unseen super-population, we present formulas for analyzing evaluation data, measuring differences between two models, and planning an evaluation experiment. We make a number of specific recommendations for running language model evaluations and reporting experiment results in a way that minimizes statistical noise and maximizes informativeness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。