用200倍少的数据，仍能准确评估大模型性能。
Rethinking LLM Evaluation: Can We Evaluate LLMs with 200x Less Data?
- 提出分步优化框架EssenceBench，结合遗传算法与样本归因分析。
- 在HellaSwag上仅用1/200样本，模型排名误差小于5%。
- 适合需要高效评测大模型能力的研究者和工程师。
随着对大模型多维度能力评估需求的增长，基准测试集规模不断扩大。尽管已有方法在冗余削减和子集性能预测方面取得进展，但缺乏系统性框架以兼顾预测精度与排序一致性。本文首先进行样本级冗余分析，识别出可删除的高相似样本；随后将基准压缩建模为分数重建优化问题。基于此，提出EssenceBench框架，采用迭代遗传算法，融合适应度驱动的子集搜索与归因驱动的样本搜索。相比现有方法，该框架实现更优压缩效果，重建误差更低、效率更高。在包含1万样本的HellaSwag基准上，仅使用25倍少的样本即可保持所有模型排名变化在5%以内；使用200倍少的样本时，仍能达到95%的排名保留率且变化在5%内。
原文摘要 · Abstract (English)
As the demand for comprehensive evaluations of diverse model capabilities steadily increases, benchmark suites have correspondingly grown significantly in scale. Despite notable advances in redundancy reduction and subset-level performance prediction, a systematic framework that effectively integrates these methods to ensure both prediction accuracy and ranking consistency is still largely elusive. In this paper, we first perform a sample-level analysis of benchmark redundancy and identify several highly similar samples that can be eliminated. Besides, we frame benchmark compression as an optimization problem with the aim of score reconstruction. Building on these, we then propose EssenceBench, a coarse-to-fine framework utilizing an iterative Genetic Algorithm (GA), which takes the advantages of fitness-based subset search and attribution-based sample search. Compared to previous methods, our approach yields superior compression results with lower reconstruction error and markedly higher efficiency. In particular, on the HellaSwag benchmark (10K samples), our method preserves the ranking of all models shifting within 5% using 25x fewer samples, and achieves 95% ranking preservation shifting within 5% using only 200x fewer samples.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。