arXiv:2605.07075cs.LG2026-05被引 3

从海量模型中自动推荐最适合任务的模型,无需实际测试。

ModelLens: Finding the Best for Your Task from Myriads of Models

论文配图:ModelLens: Finding the Best for Your Task from Myriads of Models
图 1 · 摘自论文原文
  • 利用公开排行榜数据构建模型能力隐空间,直接学习跨任务性能
  • 在162万条记录上验证,推荐效果超越依赖元数据或需实测的基线
  • 适合快速选型、模型路由优化,尤其适用于新数据集场景

开源模型生态已包含数十万预训练模型,但为新数据集选择最佳模型日益困难:新模型和未评测数据集持续涌现,从业者缺乏历史参考。现有方法仅处理此现实场景的部分问题:AutoML与迁移性估计仅限于小规模预定义模型池,或需对目标数据集进行昂贵的逐模型前向推理;模型路由则依赖给定候选池。本文提出ModelLens,一个面向真实场景的统一模型推荐框架。核心洞察是:尽管公开排行榜数据分散且嘈杂,但其集体分布可映射出跨异构评估环境的模型能力隐式图谱,蕴含足够信号用于直接学习。通过在模型-数据集-指标三元组上学习性能感知的隐空间,ModelLens可在不运行候选模型的前提下,对未见模型与未见数据集进行排序。在包含162万条评估记录、4.7万模型与9.6千数据集的新基准上,ModelLens优于仅依赖元数据或需逐模型测试的基线。其推荐的Top-K候选池进一步使多个代表性路由方法在多样化问答基准上提升最高达81%。对近期发布基准的案例研究也证实其在文本与视觉语言任务上的泛化能力。

原文摘要 · Abstract (English)

The open-source model ecosystem now contains hundreds of thousands of pretrained models, yet picking the best model for a new dataset is increasingly infeasible: new models and unbenchmarked datasets emerge continuously, leaving practitioners with no prior records on either side. Existing approaches handle only fragments of this in-the-wild setting: AutoML and transferability estimation select models from small predefined pools or require expensive per-model forward passes on the target dataset, while model routing presupposes a given candidate pool. We introduce ModelLens, a unified framework for model recommendation in the wild. Our key insight is that public leaderboard interactions, though scattered and noisy, collectively trace out an implicit atlas of model capabilities across heterogeneous evaluation settings, a signal rich enough to learn from directly. By learning a performance-aware latent space over model--dataset--metric tuples, ModelLens ranks unseen models on unseen datasets without running candidates on the target dataset. On a new benchmark of 1.62M evaluation records spanning 47K models and 9.6K datasets, ModelLens surpasses baselines that either rely on metadata alone or require running each candidate on the target dataset. Its recommended Top-K pools further improve multiple representative routing methods by up to 81% across diverse QA benchmarks. Case studies on recently released benchmarks further confirm generalization to both text and vision-language tasks.

模型推荐AutoML零样本选择多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。