小样本评估时别用中心极限定理，会严重低估误差
Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints
- 用贝叶斯方法替代传统中心极限定理估算不确定性
- 在少于几百个样本时，CLT导致误差条过窄，结果不可靠
- 提供易用的Python库，适合实际模型评估场景
大语言模型（LLM）的严谨统计评估，包括有效的误差条和显著性检验，对性能评估的可靠性至关重要。当前报告此类统计量时普遍依赖中心极限定理（CLT）。本文指出，当基准测试包含数千个样本时，基于CLT的方法是合适的，但在小样本、高度专业化基准下，这些方法表现极差，通常大幅低估不确定性（即误差条过小）。我们提出适用于这类常见小数据场景的替代频率学与贝叶斯方法，并提供一个简单的Python库实现，地址为 https://github.com/sambowyer/bayes_evals。
原文摘要 · Abstract (English)
Rigorous statistical evaluations of large language models (LLMs), including valid error bars and significance testing, are essential for meaningful and reliable performance assessment. Currently, when such statistical measures are reported, they typically rely on the Central Limit Theorem (CLT). In this position paper, we argue that while CLT-based methods for uncertainty quantification are appropriate when benchmarks consist of thousands of examples, they fail to provide adequate uncertainty estimates for LLM evaluations that rely on smaller, highly specialized benchmarks. In these small-data settings, we demonstrate that CLT-based methods perform very poorly, usually dramatically underestimating uncertainty (i.e. producing error bars that are too small). We give recommendations for alternative frequentist and Bayesian methods that are both easy to implement and more appropriate in these increasingly common scenarios. We provide a simple Python library for these Bayesian methods at https://github.com/sambowyer/bayes_evals .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。