快速定位大模型在动态测试中的薄弱环节。
QuickScope: Certifying Hard Questions in Dynamic LLM Benchmarks
- 用改进的贝叶斯优化算法搜索最难问题
- 比基线少用样本，更准发现真实难点
- 支持自定义数据集和评估目标，适合调试模型
大模型评测正变得动态化：不再固定题目，而是通过模板和参数生成无限变体。这种灵活性虽好，但评估成本高，尤其当目标不仅是平均分，而是精准识别模型弱点时。本文提出新方法 QuickScope，基于近期贝叶斯优化算法 COUP（Graham, Velez & Leyton-Brown, 2026），经多项改进使其适用于实际大模型流水线。我们还开发了工具，支持灵活选择数据集与效用函数，可针对性寻找低准确率问题或复杂度异常高的难题。在多个基准测试中，QuickScope 比标准基线更高效地发现真正难例，同时降低由噪声导致的误报。
原文摘要 · Abstract (English)
LLM benchmarks are increasingly dynamic: instead of containing a fixed set of questions, they define templates and parameters that can generate an effectively unlimited number of question variants. This flexibility is valuable, but it makes evaluation expensive -- especially when the goal is not just determining an average score, but reliably identifying a model's weak spots. This paper introduces a new methodology for identifying hard questions in dynamic benchmarks. It leverages COUP, a recent Bayesian optimization algorithm (Graham, Velez & Leyton-Brown, 2026), after introducing several substantive modifications to make the algorithm suitable for practical LLM pipelines. We also wrap it in a tool that supports flexible choices of datasets and utility functions, enabling users to target the kinds of questions they care about (e.g., low-accuracy questions; questions that are unusually hard relative to their measured complexity). In experiments across a range of benchmarks, we show that our method, dubbed $\texttt{QuickScope}$, discovers truly difficult questions more sample efficiently than standard baselines, while also reducing false positives from noisy outcomes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。