用子模优化选出少量高信息量评测集，节省大模型评估成本
Submodular Benchmark Selection
- 基于高斯模型将评测集选择建模为子模最大化问题
- 互信息目标在小样本下表现优于熵，提升评估精度
- 适用于需要高效评测大模型的科研与工程人员
在多个基准上评估大语言模型成本高昂，但许多基准高度相关。本文将选取少量有代表性基准的问题形式化为多变量高斯模型下的子模最大化问题。以熵（协方差对数行列式）和所选与剩余基准间的互信息作为自然目标函数，二者均为子模函数。熵选择等价于回溯乔列斯基分解，具有谱残差界；互信息一般非单调，但在小样本下经验上单调，因此采用贪心优化。在来自十个公开排行榜的三组矩阵上实验表明，互信息选择在小样本时的插补性能优于熵。
原文摘要 · Abstract (English)
Evaluating large language models across many benchmarks is expensive, yet many benchmarks are highly correlated. We formalize the selection of a small, informative subset as submodular maximization under a multivariate Gaussian model. Entropy (log-determinant covariance) and mutual information between selected and remaining benchmarks arise as natural objectives. Both are submodular; entropy selection coincides with pivoted Cholesky and has spectral residual bounds, while mutual information is non-monotone in general but empirically monotone for small subsets, so we optimize it greedily. Experiments on three matrices from ten public leaderboards show that mutual information selection outperforms entropy for imputation at small subsets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。