arXiv:2606.29667cs.CVcond-mat.mtrl-sci2026-06

将材料科学文献中的复杂图分解为图文对,构建可直接用于AI训练的数据集。

MatMMExtract: An Open-Source Pipeline for Panel-Level Extraction of Grounded Image-Text Pairs from Materials Science Literature

论文配图:MatMMExtract: An Open-Source Pipeline for Panel-Level Extraction of Grounded Image-Text Pairs from Materials Science Literature
图 1 · 摘自论文原文
  • 通过大模型与领域知识库,自动拆分复合图表并生成精准图文标注。
  • 从1.48万篇论文中提取39万组面板级图文对,涵盖19类可视化类型。
  • 提供专用检测数据集和开源工具链,适合研究者快速构建视觉语言模型。

材料科学文献中蕴藏数十年实验知识,但其图像信息长期无法被人工智能大规模利用。核心难点在于多数科学图表为复合结构,单个图注同时描述多个子图,导致直接图文配对不可靠。我们提出MatMMExtract,一个端到端开源流程,通过将复合图分解为独立子图,并利用大语言模型结合精心设计的材料科学分类体系,生成结构化、有依据的标注。该方法应用于14,810篇开放获取论文,生成了包含391,606个面板级图文对的MatSciFig数据集,覆盖180,571张图表,每对均附有子图注、两级可视化类别(19类超100子类)及科学摘要。为实现精准子图定位,我们构建了包含2,811张人工标注材料图的MaterialScope数据集,经微调的YOLO12-m检测器在mAP_50上达到0.9227。在六种对比语言模型中,Gemini 3.1 Flash Lite表现最优,82%输出被评为优质,幻觉率仅4.8%。基于MatSciFig的双编码器检索基线相比零样本CLIP在R@1上提升4.4倍,证明该数据集对视觉语言学习的即时价值。所有资源均已公开发布。

原文摘要 · Abstract (English)

The materials science literature encodes decades of experimental knowledge in figures, yet this visual record remains locked away and inaccessible to AI at scale. The core difficulty is structural: most scientific figures are compound, with a single caption describing multiple sub-panels simultaneously, making direct image-text pairing unreliable. We present MatMMExtract, an end-to-end open-source pipeline that resolves this by decomposing compound figures into individual sub-panels and generating structured, grounded annotations using a large language model guided by a curated materials science taxonomy. Applied to 14,810 open-access articles, MatMMExtract produces MatSciFig; 391,606 panel-level image-text pairs from 180,571 figures, each annotated with a sub-caption, a two-level visualisation category spanning 19 classes and over 100 subtypes, and a scientific summary. To enable accurate panel localisation, we introduce MaterialScope, a domain-specific detection dataset of 2,811 manually annotated materials science figures, on which a fine-tuned YOLO12-m detector achieves mAP_50 of 0.9227. Among six benchmarked language models, Gemini 3.1 Flash Lite delivers the best cost-quality trade-off for annotation generation, with 82% of outputs rated good and a hallucination rate of 4.8%. A dual-encoder retrieval baseline on MatSciFig achieves a 4.4 times improvement in R@1 over zero-shot CLIP, demonstrating the dataset's immediate utility for vision-language learning. All resources are released openly to the community.

材料科学图文对数据集开源工具

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。