用少量标注选最优大模型，效率提升超80%
Large Language Model Selection with Limited Annotations
- 基于输出相似性计算信息增益，主动筛选最有价值的测试样本
- 在23个数据集上实现最高81.8%的标注成本降低
- 无需模型权重，适用于开源与闭源大模型
为特定任务选择大语言模型需比较大量优秀候选模型，但传统评估依赖昂贵的人工标注和固定评估集。为此，我们提出SELECT-LLM——首个用于大模型主动选择的框架。该框架旨在找出一组最能帮助识别最优模型的查询样本。通过基于成对模型输出相似性的期望信息增益规则，仅使用模型生成结果即可完成查询选择，无需假设模型架构或访问模型权重，适用于开箱即用及黑盒大模型。我们在23个数据集、156个模型、多种任务类型及多个文本评估指标下验证了SELECT-LLM。所有实验中，该方法均优于最强基线，在最佳模型选择上标注成本降低最高达81.8%，在近优模型选择上降低最高达84.78%。
原文摘要 · Abstract (English)
Choosing a Large Language Model (LLM) for a given task requires comparing many strong candidates, yet standard evaluation relies on costly annotations over fixed evaluation sets. To address this challenge, we develop SELECT-LLM, the first framework for active model selection of LLMs. SELECT-LLM aims to find a small set of queries whose annotations are most informative for identifying the best LLM for a given task. To this end, we introduce a query selection rule based on expected information gain, computed from pairwise similarities between candidate model outputs. Because this rule only uses generated model responses, SELECT-LLM can be applied across candidate models without assumptions about their architecture or access to model weights. This makes it suitable for both open-weight and black-box LLMs. We evaluate SELECT-LLM across 23 datasets, 156 evaluated models, diverse task families, and multiple text evaluation metrics. Across all experiments, SELECT-LLM improves over the strongest baseline in every setting, with annotation cost reductions up to 81.8% for best model selection and up to 84.78% for near-best model selection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。