无需标注数据，快速评估新模型性能。
Learning to Evaluate: Cost-Effective Model Evaluation on Unlabeled Data with Meta-Learning
- 通过元学习从参考模型中提取评估初始化。
- 在无标签数据上实现高精度性能估计，成本降低显著。
- 适用于各类模型架构，适合大规模模型评测。
机器学习的快速发展导致模型生态急剧扩张，对未见、无标签数据上的模型可靠性评估变得日益困难。现有评估流程通常依赖昂贵的人工标注、重复微调，或依赖难以泛化的假设。我们提出 MetaEvaluator，一种低成本、模型无关的框架，可在多样架构和模态下对未见模型进行快速、无标签评估。MetaEvaluator 在参考模型池上进行元学习，获得可用于准确评估新模型的有效初始化，从而分摊评估成本，避免针对每个模型重新训练。据我们所知，这是首个在无标签数据上评估新模型的模型无关框架。大量实验表明，该方法在显著更低的成本下，仍能提供稳定且准确的性能估计，支持新兴模型在无标签数据上的可扩展基准测试。代码已开源：https://github.com/phkhanhtrinh23/MetaEvaluator。
原文摘要 · Abstract (English)
The rapid advancement of machine learning has led to an unprecedented expansion of model ecosystems, making it increasingly difficult to assess the reliability of newly released models on unseen and unlabeled data. Existing evaluation pipelines typically rely on costly annotation, repeated fine-tuning, or assumptions that do not generalize well to new models. We introduce MetaEvaluator, a cost-effective, model-agnostic framework for fast, label-free evaluation of unseen models across diverse architectures and modalities. MetaEvaluator meta-learns over a pool of reference models to acquire an effective initialization for accurate assessment of unseen models, thereby amortizing evaluation cost and eliminating the need for per-model retraining. To the best of our knowledge, this is the first model-agnostic framework that evaluates new models on unlabeled datasets. Extensive experiments demonstrate that MetaEvaluator delivers stable and accurate performance estimates at substantially lower cost than conventional approaches, enabling scalable benchmarking on unlabeled datasets for emerging models. The code is available at: https://github.com/phkhanhtrinh23/MetaEvaluator.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。