用图模型自动发现模型与数据集的最佳组合，助力科研突破。
ArtifactLinker: Linking Scientific Artifacts for Automatic State-of-the-Art Discovery
- 构建模型与数据集的关联图谱，用图神经网络预测潜在最优搭配。
- 在14,053个科学资源、51,337条关系上验证，显著提升SOTA发现效率。
- 适合关注自动化研究发现、模型评估优化的研究者使用。
科学模型和数据集是研究的基础。随着HuggingFace等平台的快速发展，研究人员可获取大量科学资源，但如何自动发现特定数据集上的最先进（SOTA）模型仍是一大挑战。本文将该任务形式化为通过建模HuggingFace为资源图谱来实现自动SOTA发现，其中节点代表模型或数据集，边表示评估关系。提出ArtifactLinker框架，包含两个阶段：(1) 使用图神经网络（GNN）或图增强型大语言模型（LLM）对未观测到的模型-数据集链接进行排序；(2) 通过基于LLM的智能体执行编码实验验证排名靠前的链接。进一步引入基准测试集ArtifactBench，包含14,053个资源和51,337条关系，用于评估两阶段性能。结果表明：(1) 现有资源间的图结构对缺失链接预测有效；(2) 基于ArtifactLinker的端到端排序与验证能有效发现潜在的SOTA结果及研究洞见。
原文摘要 · Abstract (English)
Scientific artifacts such as models and datasets are foundations for research. With the rapid growth of platforms like HuggingFace, researchers now have access to a large number of artifacts. Yet, a key challenge remains: how can we automatically discover the state-of-the-art (SOTA) model for a given dataset by fully leveraging existing artifacts? We formalize this task as automatic SOTA discovery by modeling HuggingFace as an artifact graph, where nodes are models/datasets and edges represent evaluations. We propose ArtifactLinker, a two-stage framework: (1) ranking promising unobserved model--dataset links using Graph Neural Networks (GNNs) or graph-augmented Large Language Models (LLMs), and (2) verifying top-ranked links via coding experiments with LLM-based agents. We further introduce a benchmark named ArtifactBench with 14,053 artifacts and 51,337 relations to evaluate the performance of both stages. Results show that (1) graph structures between existing artifacts are effective for missing link prediction; (2) end-to-end ranking and verification with ArtifactLinker help discover potential SOTA results and research insights.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。