用少量标注选最优大模型，效率提升近60%
Active Model Selection for Large Language Models
- 主动选择最有信息量的查询样本标注
- 在151个模型中选最佳/次优模型，节省59.62%标注成本
- 适合需要高效筛选大模型的研究者与工程师
我们提出 LLM SELECTOR，首个针对大语言模型（LLM）的主动模型选择框架。不同于依赖全标注数据集的现有评估方法，该框架在仅有限标注的情况下，即可高效识别出最适合特定任务的LLM。具体而言，对于任意任务，它自适应地选择一组最具信息量的查询进行标注，以最小化标注开销。为进一步降低人工标注成本，我们引入基于裁判模型的标注机制。在6个基准测试和151个LLM上的大量实验表明，该方法在选出最佳及次优模型时，可将标注成本降低最高达59.62%。
原文摘要 · Abstract (English)
We introduce LLM SELECTOR, the first framework for active model selection of Large Language Models (LLMs). Unlike prior evaluation and benchmarking approaches that rely on fully annotated datasets, LLM SELECTOR efficiently identifies the best LLM with limited annotations. In particular, for any given task, LLM SELECTOR adaptively selects a small set of queries to annotate that are most informative about the best model for the task. To further reduce annotation cost, we leverage a judge-based oracle annotation model. Through extensive experiments on 6 benchmarks with 151 LLMs, we show that LLM SELECTOR reduces annotation costs by up to 59.62% when selecting the best and near-best LLM for the task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。