arXiv:2606.27997cs.LGstat.ML2026-06KDD

选出少量数据集就能准确保持模型排名,提升评测效率。

Benchmarking on Tasks That Matter: Dataset Selection for Preserving Model Rankings

论文配图:Benchmarking on Tasks That Matter: Dataset Selection for Preserving Model Rankings
图 1 · 摘自论文原文
  • 用统计方法评估不同数据集选法对模型排名的影响
  • 仅用5个数据集就实现0.95的排名相关性(时序分类)
  • 适合需要高效模型评测的研究者和工程团队

机器学习模型评测常涉及大量数据集,导致评估成本高昂。为提高效率,应选用少量代表性数据集进行评测。现有数据集子集选择多依赖启发式方法,且缺乏对模型排名鲁棒性的分析。本文提出一个框架,用于评估不同选择策略在保留全局模型排名方面的表现。该框架引入自助聚合(bootstrap aggregation),提供有效置信区间,支持策略间的严谨比较。我们考察了聚类、设计准则(A/D-最优)、随机基线及贪心最远优先(FAFI)等方法。针对FAFI,我们推导了排名误差上限,其随选中数据集数量变化。实验表明,在时序分类(TSC,112个数据集)和基于MTEB的自然语言处理基准(57项任务)中,多种策略优于随机抽样,包括简单高效的FAFI;而在推荐系统(30个数据集)中,改进不显著。最佳策略仅用5个数据集即达到0.95的斯皮尔曼相关性。结果还显示,方法有效性取决于数据集表示质量和评测规模。

原文摘要 · Abstract (English)

Benchmarks of machine learning models often include many datasets, making evaluation expensive. For efficiency, it is preferable to perform evaluations on small, representative datasets instead. The selection of such subsets typically relies on heuristics and is rarely analyzed for the robustness of the resulting model rankings. We introduce a framework to perform the task of selecting datasets subsets with an evaluation of how different selection strategies preserve the global model rankings. Our framework includes bootstrap aggregation, which provides valid confidence intervals, allowing a principled comparison of selection strategies. We consider clustering, design criteria (A/D-optimality), random baselines, and greedy farthest-first (FAFI). For the latter, we derive upper bounds on selection quality in terms of ranking errors as a function of the number of selected datasets. Empirically, in time series classification (TSC, 112 datasets) and in a supplementary natural language processing benchmark derived from MTEB (57 tasks), several selection strategies improve rank preservation compared with random subsets, including simple FAFI. In contrast, in recommender systems (30 datasets), the improvement of strategies over random selection is small and typically statistically insignificant. For TSC, our best-performing strategy achieves a Spearman correlation of 0.95 with the full benchmark model rankings using only five selected datasets. Additional experiments indicate that the effectiveness of selection approaches depends on both the quality of dataset representations and the scale of the benchmarking regime.

模型评测数据集筛选排名保持高效评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。