根据不确定性动态停止评估，大幅减少LLM测试计算量。
Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations
- 基于贝叶斯推理的自适应采样，按不确定性决定是否继续测试
- 在200个任务、10轮评估中减少57%~97%测试次数，结果与全量测试一致
- 特别适合高精度要求或罕见成功场景，节省大量算力
LLM评估通常采用固定采样预算，对每个项目重复相同次数，即使估计已足够精确。我们提出optstop，一种基于精度的自适应停止框架，将评估视为顺序测量问题：在不确定性高的地方持续采样，在估计足够精确或稳定时停止。该框架基于分层贝叶斯推断，支持二分类、有序和连续结果，所有基准项目均可参与采样，无需校准的项目库。可实时或回溯运行，并包含安全机制——当性能接近零时更谨慎采样，以捕捉罕见成功。在200项任务、10轮评估的示例中，九种验证设置下移除了57%~97%的计划测试，整体结论与完整运行等效。结果表明，可通过不确定性分配评估算力，而非固定重复次数，节省幅度取决于评估设计。
原文摘要 · Abstract (English)
LLM evaluations often use fixed sampling budgets, testing every item the same number of times even after estimates are precise. We introduce optstop, a precision-based adaptive stopping framework that treats evaluation as a sequential measurement problem: keep sampling where uncertainty remains high, and stop where estimates are precise or stable enough. The framework builds on hierarchical Bayesian inference, supports binary, ordinal, and continuous outcomes, and keeps every benchmark item eligible for sampling, without requiring a calibrated item bank. It runs live or retrospectively, and includes a safeguard that samples more cautiously as measured performance approaches zero, where rare successes matter most. In an illustrative 200-item, 10-epoch evaluation, it removes 57%-97% of planned trials across nine validation settings, with overall conclusions equivalent to the full run. These results show that LLM evaluation compute can be allocated by uncertainty rather than by fixed repetition counts, with the magnitude of savings depending on evaluation design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。