从模型仓库中挖掘被忽视的高性能微调模型。
Discovering Hidden Gems in Model Repositories
- 将模型发现建模为多臂赌博机问题，用共享查询集加速搜索。
- 仅需每候选50次查询即可找到顶尖模型，提速超50倍。
- 发现冷门模型在数学能力上提升12.8个百分点，成本不变。
公共模型仓库中存放着数百万个微调模型，但社区使用却高度集中在少数基础模型上。我们探究这种集中是高效市场选择的结果，还是优质模型被系统性忽视。通过对超过2000个模型的广泛评估，我们发现大量‘隐藏珍宝’——即不受欢迎但显著优于主流模型的微调版本。特别地，在Llama-3.1-8B系列中，一些极少下载的检查点将数学性能从83.2%提升至96.0%，且推理成本未增加。然而，对每个上传模型进行穷尽评估在计算上不可行。因此，我们将模型发现建模为多臂赌博机问题，并通过共享查询集和激进淘汰策略加速顺序二分搜索算法。我们的方法使每候选仅需约50次查询即可检索到顶级模型，发现速度提升超过50倍。
原文摘要 · Abstract (English)
Public repositories host millions of fine-tuned models, yet community usage remains disproportionately concentrated on a small number of foundation checkpoints. We investigate whether this concentration reflects efficient market selection or if superior models are systematically overlooked. Through an extensive evaluation of over 2,000 models, we show the prevalence of "hidden gems", unpopular fine-tunes that significantly outperform their popular counterparts. Notably, within the Llama-3.1-8B family, we find rarely downloaded checkpoints that improve math performance from 83.2% to 96.0% without increasing inference costs. However, discovering these models through exhaustive evaluation of every uploaded model is computationally infeasible. We therefore formulate model discovery as a Multi-Armed Bandit problem and accelerate the Sequential Halving search algorithm by using shared query sets and aggressive elimination schedules. Our method retrieves top models with as few as 50 queries per candidate, accelerating discovery by over 50x.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。