用缓存结果预测新模型性能，大幅降低评测查询次数。
Query-efficient model evaluation using cached responses
- 基于数据核空间方法，利用旧模型缓存响应推断新模型表现。
- 在相同误差下，所需查询量比基线减少显著，实测效果优异。
- 可离线选最优查询集，适合资源受限的模型评测场景。
在现有基准上评估新模型以了解其行为是部署前的必要步骤。现代评估框架中，对所有查询生成并评估响应成本高昂。实践中，以往模型的响应常被缓存，这为减少新模型评测所需查询数提供了可能。本文提出一种基于数据核空间（DKPS）的方法，利用缓存响应预测基准性能。理论上，证明了在特定条件下DKPS方法具有查询效率优势；实验表明，该方法在保持与基线相同平均绝对误差的前提下，显著降低查询预算。最后，提出一种离线查询集选择方法，通过最大化参考模型的拟合优度，提升预测精度，优于随机选择。
原文摘要 · Abstract (English)
Evaluating a new model on an existing benchmark is often necessary to understand its behavior before deployment. For modern evaluation frameworks, generating and evaluating a response for all queries can be prohibitively expensive. In practice, responses from previously-evaluated models are often cached -- creating a potential opportunity to use this additional information to decrease the number of queries required to accurately evaluate a new model. In this paper, we introduce an approach for predicting benchmark performance that leverages cached model responses based on the Data Kernel Perspective Space (DKPS), a method for quantifying the relationship between models in the black-box setting. Theoretically, we show that DKPS-based methods are query-efficient under certain conditions. Empirically, we demonstrate that DKPS-based methods achieve the same mean absolute error as baselines with a substantially decreased query budget. We conclude by proposing an offline method for selecting a set of queries that maximizes the goodness-of-fit on reference models, improving prediction accuracy over random query selection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。