通过利用模型相似性，用更少评估次数找出最优大模型。
Cutting LLM Evaluation Costs with SySRs: A Bandit Algorithm that Provably Exploits Model Similarity
- 用配对比较增强经典淘汰算法，自适应分配评估资源。
- 在15个基准上平均误差率更低，最差情况也更省预算。
- 无需调参且性能随模型相似度提升而增强，适合部署选型。
大型语言模型通常需在每个测试查询上评估所有模型，这对实践者而言常为浪费：若某模型明显劣于其他，则无需精确估算其表现。最佳臂识别算法可自然用于大幅降低评估成本，通过自适应分配评估预算实现。此外，语言模型对同一提示的响应往往相似——此前研究尝试利用此特性但效果参差。本文提出同步连续淘汰（SySRs），在经典连续淘汰算法基础上引入配对比较。与以往利用模型相似性的方法不同，该方法无需超参数调整，且性能保证随模型间相似度提升而改善。实验表明，该方法在15个标准基准上的平均误差率均低于所有基线，且在可靠识别最优模型的最差情况预算上也表现更优。
原文摘要 · Abstract (English)
Large Language Models are typically benchmarked by evaluating every model on every test query. For practitioners seeking the best model to deploy, this is often wasteful: if a model clearly performs worse than others, there is no need to precisely estimate its performance. Best-arm identification algorithms can be naturally applied to drastically reduce costs by adaptively allocating evaluation budget. Further, language models often respond similarly to the same prompt-a property previous work has tried to leverage with mixed success. We propose Synchronized Successive Rejects (SySRs), augmenting the classical Successive Rejects algorithm with paired comparisons. Unlike prior attempts to leverage model similarity in best-model identification, our approach is hyperparameter-free and enjoys performance guarantees that improve with the degree of similarity between evaluated models. Empirically, our method outperforms all baselines in terms of average error rate across 15 standard benchmarks, and in terms of worst-case budget for reliably identifying the best model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。