构建首个大规模模型表格基准,助力精准查找模型性能数据。
ModelTables: A Corpus of Tables about Models
- 从模型卡片、README 和论文中提取结构化表格数据
- 覆盖超6万模型与9万张表格,验证表间关联性更强
- 适合研究模型知识组织与表格搜索的学者使用
我们提出ModelTables,一个来自Hugging Face模型卡片、GitHub README和参考论文的表格语料库,捕捉常被文本检索忽略的模型性能与配置表格的结构化语义。该语料库链接每张表格与其对应的模型及出版物上下文,相比开放数据湖中的表格,模型表格规模更小但表间关系更密集,反映模型与评测体系的紧密演化。当前版本涵盖超过60,000个模型和90,000张表格。为评估模型与表格相关性,我们基于三种互补信号构建多源真值:(1)论文引用链接,(2)显式模型卡片链接与继承关系,(3)共享训练数据集。我们展示了该基准的一个典型应用场景——表格搜索,并对比了标准数据湖搜索算子(可并集、可连接、关键词)与信息检索基线(密集、稀疏、混合检索)的表现。基于并集的语义表格检索在整体上达到54.8% P@1(引用信号54.6%,继承信号31.3%,共享数据信号30.6%),基于表格的密集检索达到66.5% P@1,元数据混合检索为54.1%。结果表明,现有表格搜索方法仍有显著提升空间。通过发布ModelTables及其构建流程,我们首次提供了大规模描述人工智能模型的结构化数据基准。其在模型湖中发现表格的应用案例,为开发更准确的语义检索、结构化比较与模型知识的系统化组织提供了依据。源代码、数据及其他资源已公开于https://github.com/RJMillerLab/ModelTables。
原文摘要 · Abstract (English)
We present ModelTables, a benchmark of tables in Model Lakes that captures the structured semantics of performance and configuration tables often overlooked by text only retrieval. The corpus is built from Hugging Face model cards, GitHub READMEs, and referenced papers, linking each table to its surrounding model and publication context. Compared with open data lake tables, model tables are smaller yet exhibit denser inter table relationships, reflecting tightly coupled model and benchmark evolution. The current release covers over 60K models and 90K tables. To evaluate model and table relatedness, we construct a multi source ground truth using three complementary signals: (1) paper citation links, (2) explicit model card links and inheritance, and (3) shared training datasets. We present one extensive empirical use case for the benchmark which is table search. We compare canonical Data Lake search operators (unionable, joinable, keyword) and Information Retrieval baselines (dense, sparse, hybrid retrieval) on this benchmark. Union based semantic table retrieval attains 54.8 % P@1 overall (54.6 % on citation, 31.3 % on inheritance, 30.6 % on shared dataset signals); table based dense retrieval reaches 66.5 % P@1, and metadata hybrid retrieval achieves 54.1 %. This evaluation indicates clear room for developing better table search methods. By releasing ModelTables and its creation protocol, we provide the first large scale benchmark of structured data describing AI model. Our use case of table discovery in Model Lakes, provides intuition and evidence for developing more accurate semantic retrieval, structured comparison, and principled organization of structured model knowledge. Source code, data, and other artifacts have been made available at https://github.com/RJMillerLab/ModelTables.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。