用图遍历模拟人类思考,让模型更准识别分子结构
GTR-CoT: Graph Traversal as Visual Chain of Thought for Molecular Structure Recognition
- 通过逐步预测原子键来模拟人看分子图的推理过程
- 手绘分子识别准确率显著提升,超越现有模型
- 适合化学、药物发现领域研究人员参考
光学分子结构识别(OCSR)对将分子图像转为机器可读格式至关重要。尽管近期视觉语言模型(VLMs)表现良好,但其图像-标题生成方式在复杂分子结构和标注不一致时效果不佳。为此,我们提出GTR-VL,包含两项创新:(1)图遍历作为视觉思维链机制,通过逐次预测原子-键来模拟人类逐步解析分子图的推理过程;(2)数据导向的“所见即所识”原则,使图像中的简写结构与扩展标注对齐。针对手绘OCSR任务中缺乏图标注、仅提供最终SMILES的情况,我们采用GRPO强化学习方法,引入格式奖励、图结构奖励和SMILES奖励,实现弱监督下的性能提升。我们构建了包含修正标注的大规模指令微调数据集GTR-1.3M,以及首个用于细粒度图解析准确率评估的基准MolRec-Bench。采用两阶段训练策略:先对印刷体图像进行SFT训练,再通过GRPO方法迁移至手绘任务。实验表明,GTR-VL在印刷体和手绘数据集上均优于专用模型、化学领域VLM及商用VLM。
原文摘要 · Abstract (English)
Optical Chemical Structure Recognition (OCSR) is essential for converting molecular images into machine-readable formats. While recent vision-language models (VLMs) have shown promise, their image-captioning approach often struggles with complex molecular structures and inconsistent annotations. To address these issues, we introduce GTR-VL, featuring two key innovations: (1) the \textit{Graph Traversal as Visual Chain of Thought} mechanism that emulates human reasoning by incrementally parsing molecular graphs through sequential atom-bond predictions, and (2) the data-centric \textit{Faithfully Recognize What You've Seen} principle, which aligns abbreviated structures in images with their expanded annotations. For hand-drawn OCSR tasks, where datasets lack graph annotations and only provide final SMILES, we apply reinforcement learning using the GRPO method, introducing reward mechanisms like format reward, graph reward, and SMILES reward. This approach significantly enhances performance in hand-drawn recognition tasks through weak supervision. We developed GTR-1.3M, a large-scale instruction-tuning dataset with corrected annotations, and MolRec-Bench, the first benchmark for fine-grained evaluation of graph-parsing accuracy in OCSR. Our two-stage training scheme involves SFT training for printed images and the GRPO method for transferring capabilities to hand-drawn tasks. Experiments show that GTR-VL outperforms specialist models, chemistry-domain VLMs, and commercial VLMs on both printed and hand-drawn datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。