用稀疏优化加速大模型评估，降低计算成本。
SparseEval: Efficient Evaluation of Large Language Models by Sparse Optimization
- 通过梯度下降优化关键样本权重，迭代筛选代表性测试项。
- 在多个基准上误差低、相关性高，评估结果稳定可靠。
- 适合需要高效评测大模型性能的研究者和工业应用。
随着大语言模型（LLMs）规模持续扩大，其在各类下游任务上的表现显著提升。然而，对模型能力的评估成本也随之上升，因在大量基准样本上进行推理带来高昂的计算开销。本文重新审视模型-项目性能矩阵，发现其具有稀疏性：可通过选择代表性项目作为锚点实现高效评估，并将该问题建模为稀疏优化问题。基于此，我们提出 SparseEval，首次采用梯度下降优化锚点权重，并引入迭代精炼策略进行锚点选择。利用MLP的表征能力处理稀疏优化，提出锚点重要性评分与候选重要性评分，以评估各测试项在任务感知下的价值。大量实验表明，该方法在多种基准上均保持低估计误差与高Kendall's τ，展现出卓越的鲁棒性与实际应用价值。代码已公开于 https://github.com/taolinzhang/SparseEval。
原文摘要 · Abstract (English)
As large language models (LLMs) continue to scale up, their performance on various downstream tasks has significantly improved. However, evaluating their capabilities has become increasingly expensive, as performing inference on a large number of benchmark samples incurs high computational costs. In this paper, we revisit the model-item performance matrix and show that it exhibits sparsity, that representative items can be selected as anchors, and that the task of efficient benchmarking can be formulated as a sparse optimization problem. Based on these insights, we propose SparseEval, a method that, for the first time, adopts gradient descent to optimize anchor weights and employs an iterative refinement strategy for anchor selection. We utilize the representation capacity of MLP to handle sparse optimization and propose the Anchor Importance Score and Candidate Importance Score to evaluate the value of each item for task-aware refinement. Extensive experiments demonstrate the low estimation error and high Kendall's~$τ$ of our method across a variety of benchmarks, showcasing its superior robustness and practicality in real-world scenarios. Code is available at {https://github.com/taolinzhang/SparseEval}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。