用统计方法高效评估大模型性能，少用测试题也能保证结果可信。
Efficient Evaluation of LLM Performance with Statistical Guarantees
- 基于历史数据构建贝叶斯因子模型，智能选题减少冗余测试。
- 在相同测试次数下，置信区间宽度比基线缩小最多5倍。
- 适合需要快速、可靠评估多个大模型的研究者使用。
在大量基准测试上全面评估众多大语言模型成本高昂。本文将评测问题建模为有限总体推断，在固定查询预算下，目标是获得具有有效频率覆盖的模型准确率置信区间。提出因子化主动查询（FAQ）方法：(a) 通过贝叶斯因子模型利用历史信息；(b) 采用混合方差缩减与主动学习的采样策略自适应选择题目；(c) 通过主动推断的有限总体扩展——主动主动推断（Proactive Active Inference），实现直接选题的同时保持覆盖率。在两个基准测试套件上，FAQ以可忽略的额外开销，相比强基线实现最多5倍的有效样本量提升，即在达到相同置信区间宽度时，最多可减少5倍查询次数。我们已开源代码和整理的数据集，以支持可复现评估与后续研究。
原文摘要 · Abstract (English)
Exhaustively evaluating many large language models (LLMs) on a large suite of benchmarks is expensive. We cast benchmarking as finite-population inference and, under a fixed query budget, seek tight confidence intervals (CIs) for model accuracy with valid frequentist coverage. We propose Factorized Active Querying (FAQ), which (a) leverages historical information through a Bayesian factor model; (b) adaptively selects questions using a hybrid variance-reduction/active-learning sampling policy; and (c) maintains validity through Proactive Active Inference -- a finite-population extension of active inference (Zrnic & Candès, 2024) that enables direct question selection while preserving coverage. With negligible overhead cost, FAQ delivers up to $5\times$ effective sample size gains over strong baselines on two benchmark suites, across varying historical-data missingness levels: this means that it matches the CI width of uniform sampling while using up to $5\times$ fewer queries. We release our source code and our curated datasets to support reproducible evaluation and future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。