用上下文学习评估任务可迁移性，5%任务即可逼近全量评测效果。
BenTo: Benchmark Task Reduction with In-Context Transferability
- 通过上下文学习估算任务间可迁移性，构建高效评估指标。
- 将MMLU等基准任务压缩至5%，误差低于4%。
- 无需训练与梯度计算，适合快速评估大模型性能。
评估大语言模型（LLM）成本高昂，需在大规模多任务基准上生成并分析输出。本文研究如何在不降低评估质量的前提下，高效缩减评测任务数量。研究表明，任务间的可迁移性与相关性是识别最具代表性任务子集的关键，可通过优化设施位置函数实现。我们提出一种基于上下文学习（ICL）的实用高效指标，用于估计两任务间的可迁移性。通过分析任务对的可迁移性，可在现代LLM基准（如MMLU或FLAN）上将任务量减少至5%，同时评估结果与原基准差异小于4%。相比已有方法，本方法无需训练和梯度计算，仅依赖ICL，效率极高。
原文摘要 · Abstract (English)
Evaluating large language models (LLMs) is costly: it requires the generation and examination of LLM outputs on a large-scale benchmark of various tasks. This paper investigates how to efficiently reduce the tasks used to benchmark LLMs without affecting the evaluation quality. Our study reveals that task transferability and relevance provide critical information to identify the most representative subset of tasks via optimizing a facility location function. We propose a practically efficient metric for estimating the transferability between two tasks via in-context learning (ICL). By analyzing the pairwise transferability, we can reduce tasks in a modern LLM benchmark (e.g., MMLU or FLAN) to 5% while inducing only a <4% difference to the evaluation on the original benchmark. Compared to prior works, our method is training-free, gradient-free, and highly efficient requiring ICL only.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。