用任务本身认知难度选数据,让大模型评测更省成本且更快上手。
Scales++: Compute Efficient Evaluation Subset Selection with Cognitive Scales Embeddings
- 基于任务内在认知难度而非模型表现选样本,实现更高效的评测。
- 仅用0.25%数据预测全基准得分,误差仅3.2%,2.0%样本时误差2.9%。
- 适合快速评估新模型、冷启动场景,结果更可解释。
大规模语言模型在完整基准上的评估成本高昂,亟需小而具代表性的数据子集(即微型基准)实现高效评估并保持预测准确性。现有方法多采用以模型为中心的范式,根据已有模型的整体表现选择评测项,但存在前期成本高、无法即时支持新基准(冷启动问题),且假设未来模型会延续旧模型的失败模式等局限。本文提出一种全新的以任务为中心的评测子集选择方法,主张依据任务本身的内在属性进行选择。我们通过Scales++方法实现该思路,基于基准样本的认知需求进行数据筛选。实验表明,Scales++将前期选样成本降低18倍以上,同时保持良好预测精度:在Open LLM Leaderboard上,仅用0.25%的数据子集,即可实现3.2%的平均绝对误差;在Humanity's Last Exam上,使用2.0%的样本,预测误差为2.9%。该方法在保证评测准确性的前提下,显著提升效率,具备更好的冷启动能力与可解释性。
原文摘要 · Abstract (English)
The prohibitive cost of evaluating large language models (LLMs) on comprehensive benchmarks necessitates the creation of small yet representative data subsets (i.e., tiny benchmarks) that enable efficient assessment while retaining predictive fidelity. Current methods for this task operate under a model-centric paradigm, selecting benchmarking items based on the collective performance of existing models. Such approaches are limited by large upfront costs, an inability to immediately handle new benchmarks ("cold-start"), and the fragile assumption that future models will share the failure patterns of their predecessors. In this work, we propose a new item-centric approach to benchmark subset selection, arguing that selection should be based on the intrinsic properties of the task items themselves, rather than on model-specific failure patterns. We instantiate this item-centric efficient benchmarking approach via a novel method, Scales++, where data selection is based on the cognitive demands of the benchmark samples. Empirically, we show Scales++ reduces the upfront selection cost by over 18x while achieving competitive predictive fidelity. On the Open LLM Leaderboard, using just a 0.25% data subset, we predict full benchmark scores with a 3.2% mean absolute error, and on Humanity's Last Exam we predict full scores with 2.9% mean absolute error using a 2.0% sample. We demonstrate that this item-centric approach enables more efficient model evaluation without significant fidelity degradation, while also providing better cold-start performance and more interpretable benchmarking.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。