用连续评分改进大模型评估，少测几题也能可靠排序。
Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scores
- 将传统测试的二元判断改为连续评分，用正态分布建模误差。
- 仅用2%题目就提升排名相关性0.12 τ，95%预测可信。
- 适合追求高效精准模型比较的研究者和评测团队。
计算机自适应测试（CAT）在多项选择型基准上已证明能高效评估大语言模型，但现代大模型评估越来越多依赖生成任务，输出需连续评分而非简单对错判断。本文提出一种基于项目反应理论（IRT）的拓展方法，将伯努利响应分布替换为异方差正态分布，实现对连续有界评分（如ROUGE、BLEU、LLM-as-a-Judge）的建模。在此基础上，设计了一种带不确定性的自适应排名器，结合自适应停止准则，在最少测试项数下完成高可靠性模型排序。我们在五个涵盖n-gram、嵌入式及大模型评分指标的基准上验证了该方法：仅使用2%的测试样本，相比随机采样使排名相关性提升0.12 τ，且自信预测准确率达95%。
原文摘要 · Abstract (English)
Computerized Adaptive Testing (CAT) has proven effective for efficient LLM evaluation on multiple-choice benchmarks, but modern LLM evaluation increasingly relies on generation tasks where outputs are scored continuously rather than marked correct/incorrect. We present a principled extension of IRT-based adaptive testing to continuous bounded scores (ROUGE, BLEU, LLM-as-a-Judge) by replacing the Bernoulli response distribution with a heteroskedastic normal distribution. Building on this, we introduce an uncertainty aware ranker with adaptive stopping criteria that achieves reliable model ranking while testing as few items and as cheaply as possible. We validate our method on five benchmarks spanning n-gram-based, embedding-based, and LLM-as-judge metrics. Our method uses 2% of the items while improving ranking correlation by 0.12 τ over random sampling, with 95% accuracy on confident predictions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。