用推荐系统思路，少跑测试用例就能准估大模型真实性能。
Efficient Evaluation of Large Language Models via Collaborative Filtering
- 把模型当用户、题目当商品，选关键题来评估
- 仅需少量样本即可准确预测模型在全集上的表现
- 适合资源有限时快速对比多个大模型能力
随着大语言模型（LLMs）的发展，大量评测基准被提出以衡量和比较不同模型的能力。然而，由于测试用例数量庞大且推理速度慢，评估成本高昂。本文旨在探索如何基于从基准中采样的少量实例的评估结果，高效估算模型在该基准上的真实性能。受推荐系统中协同过滤（CF）的启发，我们将模型视为用户，测试实例视为物品，提出一种两阶段方法：第一阶段将实例选择问题视为向用户推荐商品，以挑选能有效区分模型性能的关键实例；第二阶段将性能预测问题建模为推荐系统中的评分预测任务，预测目标模型在未选实例上的表现。在多个大模型和数据集上的实验表明，该方法能在大幅降低推理开销的同时，准确估计目标模型的真实性能。
原文摘要 · Abstract (English)
With the development of Large Language Models (LLMs), numerous benchmarks have been proposed to measure and compare the capabilities of different LLMs. However, evaluating LLMs is costly due to the large number of test instances and their slow inference speed. In this paper, we aim to explore how to efficiently estimate a model's real performance on a given benchmark based on its evaluation results on a small number of instances sampled from the benchmark. Inspired by Collaborative Filtering (CF) in Recommendation Systems (RS), we treat LLMs as users and test instances as items and propose a two-stage method. In the first stage, we treat instance selection as recommending products to users to choose instances that can easily distinguish model performance. In the second stage, we see performance prediction as rating prediction problem in RS to predict the target LLM's behavior on unselected instances. Experiments on multiple LLMs and datasets imply that our method can accurately estimate the target model's performance while largely reducing its inference overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。