将化学反应图解析转化为视觉提示引导的图像描述,提升机器可读性。
RxnCaption: Reformulating Reaction Diagram Parsing as Visual Prompt Guided Captioning
- 用分子检测器预画分子框和编号,作为视觉提示输入大模型
- 在15000张真实文献图上达到当前最优结构提取效果
- 适合需要从论文中自动提取化学反应信息的研究者
大规模化学反应数据集对化学领域AI研究至关重要。然而,现有数据多以论文中的图像形式存在,无法被机器直接读取,难以用于训练模型。针对这一问题,我们提出RxnCaption框架,用于化学反应图解析(RxnDP)任务。该框架将传统的坐标预测驱动解析过程重构为图像字幕生成问题,由大视觉语言模型自然处理。我们引入一种名为BBox and Index as Visual Prompt(BIVP)的策略,利用先进的分子检测器MolYOLO,在输入图像上预先绘制分子边界框和索引,使下游任务变为自然语言描述问题。大量实验表明,BIVP策略显著提升结构提取质量,同时简化模型设计。我们进一步构建了RxnCaption-15k数据集,规模比先前真实文献基准大一个数量级,并包含跨四种布局范式的均衡测试子集。实验显示,RxnCaption-VL在多个指标上达到当前最优性能。我们认为该方法、数据集与模型将推动化学文献中结构化信息提取,促进化学领域更广泛AI应用。数据、模型与代码将在GitHub发布。
原文摘要 · Abstract (English)
Large-scale chemical reaction datasets are crucial for AI research in chemistry. However, existing chemical reaction data often exist as images within papers, making them not machine-readable and unusable for training machine learning models. In response to this challenge, we propose the RxnCaption framework for the task of chemical Reaction Diagram Parsing (RxnDP). Our framework reformulates the traditional coordinate prediction driven parsing process into an image captioning problem, which Large Vision Language Models (LVLMs) handle naturally. We introduce a strategy termed BBox and Index as Visual Prompt (BIVP), which uses our state-of-the-art molecular detector, MolYOLO, to pre-draw molecular bounding boxes and indices directly onto the input image. This turns the downstream parsing into a natural-language description problem. Extensive experiments show that the BIVP strategy significantly improves structural extraction quality while simplifying model design. We further construct the RxnCaption-15k dataset, an order of magnitude larger than prior real-world literature benchmarks, with a balanced test subset across four layout archetypes. Experiments demonstrate that RxnCaption-VL achieves state-of-the-art performance on multiple metrics. We believe our method, dataset, and models will advance structured information extraction from chemical literature and catalyze broader AI applications in chemistry. We will release data, models, and code on GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。