用心理测量学方法让大模型评估少90%题量，还能更精细区分模型性能。
Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks
- 基于项目反应理论，按模型能力动态选题，提升评估效率。
- 仅用41题即达全题库精度，0.157 MAE误差，题量减少90%。
- 能区分准确率相同模型，适合大规模模型对比与精调评估。
评估大语言模型通常需要数千个基准题目，成本高且效率低。现有方法依赖固定题集的平均准确率，忽略题目难度和区分度差异。本文提出ATLAS框架，基于项目反应理论（IRT），利用费雪信息指导题目选择，可将所需题目数量减少高达90%，同时保持测量精度。例如，在包含5,600题的HellaSwag数据集上，仅用41题即可达到0.157的平均绝对误差（MAE）水平。我们进一步从能力估计θ重构准确率，发现重构结果与原始准确率高度一致，说明θ能保留全局性能结构。同时，θ在准确率相同的模型间提供更细粒度区分：在超过3,000个模型中，23%-31%的排名变化超过10位，相同准确率的模型获得显著不同的能力估计值。代码与校准题库已开源。
原文摘要 · Abstract (English)
Evaluating large language models (LLMs) typically requires thousands of benchmark items, making the process expensive, slow, and increasingly impractical at scale. Existing evaluation protocols rely on average accuracy over fixed item sets, treating all items as equally informative despite substantial variation in difficulty and discrimination. We introduce ATLAS, an adaptive testing framework based on Item Response Theory (IRT) that estimates model ability using Fisher information-guided item selection. ATLAS reduces the number of required items by up to 90% while maintaining measurement precision. For instance, it matches whole-bank ability estimates using only 41 items (0.157 MAE) on HellaSwag (5,600 items). We further reconstruct accuracy from ATLAS's ability estimates and find that reconstructed accuracies closely match raw accuracies across all five benchmarks, indicating that ability $θ$ preserves the global performance structure. At the same time, $θ$ provides finer discrimination within accuracy-equivalent models: among more than 3,000 evaluated models, 23-31% shift by more than 10 rank positions, and models with identical accuracies receive meaningfully different ability estimates. Code and calibrated item banks are available at https://github.com/Peiyu-Georgia-Li/ATLAS.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。