用心理测量学方法大幅降低大模型缩放实验成本，只需少量测试题即可准确预测性能。
Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimation
- 将题目反应理论融入缩放定律，分离模型能力与题目特性，降低计算复杂度。
- 仅需每基准测试50题（减少99.9%），就能达到传统方法相当或更好的预测精度。
- 模型能力估计可跨任务泛化，适合快速评估多场景下大模型表现。
缩放定律为理解语言模型性能提供了基础框架，但其推导需在数千个检查点或数百万次推理样本上进行昂贵评估。为此，我们提出项目反应缩放定律（IRSL），将项目反应理论（IRT）融入缩放定律框架。不同于传统方法对每个模型-基准对独立处理，IRSL解耦了潜在模型能力与题目特征，将M个模型和N个题目的缩放定律估计复杂度从O(M×N)降至O(M+N)。我们以Beta-IRT实现该框架，利用语言模型的实证响应概率——如预训练中的词元概率、测试时采样的通过率——捕捉比二值响应更丰富的信号。我们在两种主流缩放范式中验证：(1) 预训练下游缩放，使用6,612个语言模型检查点和来自10个基准的37,682个问题；(2) 测试时缩放，使用12个模型和来自4个基准的120个问题，每题最多2,500次采样。经一次校准后，IRSL仅需每基准50个问题即可获得更可靠的缩放估计，相比传统方法效率提升99.9%，且决策准确率相当或更优。此外，估计出的潜在模型能力具有泛化性，可在共享相同测量目标的基准间实现准确性能预测。
原文摘要 · Abstract (English)
Scaling laws provide a fundamental framework for understanding the performance of Language Models (LMs), yet deriving them requires prohibitively expensive evaluations across thousands of checkpoints or millions of inference samples. To address this, we introduce Item Response Scaling Laws (IRSL), a unified framework that integrates Item Response Theory (IRT) within the scaling law framework. Unlike traditional approaches that treat each model-benchmark pair in isolation, IRSL disentangles latent model ability from question characteristics, factorizing the scaling law estimation for $M$ models and $N$ questions to significantly reduce parameter complexity from $O(M \times N)$ to $O(M + N)$. We instantiate IRSL with Beta-IRT, which leverages the empirical probability responses of LMs -- such as token probabilities in pre-training and pass rates in test-time sampling -- to capture richer signals than binary responses. We validate our approach across two prevalent scaling paradigms: (1) pre-training downstream scaling, using 6,612 LM checkpoints and 37,682 questions from 10 benchmarks; and (2) test-time scaling, using 12 LMs and 120 questions from 4 benchmarks with up to 2,500 samples per question. Given a one-time calibration on existing model responses, IRSL yields more reliable scaling estimates using only 50 questions per benchmark (a 99.9\% reduction), achieving comparable or superior decision accuracy to traditional approaches. Furthermore, we show that the estimated latent model abilities are generalizable, enabling accurate performance forecasting across benchmarks that share the same measurement objective.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。