用模型分歧选样本,让评估效率提升且更准。
DISCO: Diversifying Sample Condensation for Efficient Model Evaluation
- 按模型对样本的分歧程度选数据,不依赖复杂聚类。
- 在多个基准上预测准确率误差低于1.5%,领先现有方法。
- 适合需要快速评估大模型性能的研究者和工程团队。
现代机器学习模型的评估成本高昂,如LMMs-Eval和HELM等基准需数千小时GPU计算。高成本限制了创新速度与研究包容性,并加剧环境负担。传统方法分两步:先选锚定数据子集,再训练子集精度与最终测试结果的映射关系。但锚定选择依赖聚类,设计敏感且复杂。本文提出,真正重要的是选择能引发最大模型响应差异的样本,而非单纯保证样本多样性。DISCO方法通过贪婪策略选取使模型分歧最大的前k个样本,仅使用样本级统计量,无需全局聚类。理论分析表明,模型间分歧在信息论上是最优的贪心选择准则。实验证明,DISCO在MMLU、Hellaswag、Winogrande和ARC等多个基准上实现最优性能预测,平均误差低于1.5%。代码已开源:https://github.com/arubique/disco-public。
原文摘要 · Abstract (English)
Evaluating modern machine learning models has become prohibitively expensive. Benchmarks such as LMMs-Eval and HELM demand thousands of GPU hours per model. Costly evaluation reduces inclusivity, slows the cycle of innovation, and worsens environmental impact. The typical approach follows two steps. First, select an anchor subset of data. Second, train a mapping from the accuracy on this subset to the final test result. The drawback is that anchor selection depends on clustering, which can be complex and sensitive to design choices. We argue that promoting diversity among samples is not essential; what matters is to select samples that $\textit{maximise diversity in model responses}$. Our method, $\textbf{Diversifying Sample Condensation (DISCO)}$, selects the top-k samples with the greatest model disagreements. This uses greedy, sample-wise statistics rather than global clustering. The approach is conceptually simpler. From a theoretical view, inter-model disagreement provides an information-theoretically optimal rule for such greedy selection. $\textbf{DISCO}$ shows empirical gains over prior methods, achieving state-of-the-art results in performance prediction across MMLU, Hellaswag, Winogrande, and ARC. Code is available here: https://github.com/arubique/disco-public.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。