用思维链让大模型把科学图转成可编辑代码,无需训练
Draw with Thought: Unleashing Multimodal Reasoning for Scientific Diagram Generation
- 通过分步思维链规划,将图像转化为结构化XML代码
- 在8个大模型上实现高保真、语义对齐的重构,准确率超基准32%
- 适合需要可编辑科学图的科研人员和自动化工具开发者
科学图表是跨学科传递结构化知识的重要工具,但常以静态位图形式发表,丢失符号语义且难以复用。尽管多模态大模型为视觉与结构的融合提供可能,现有方法缺乏语义控制与结构可解释性,尤其在复杂图表上表现不佳。我们提出无需训练的Draw with Thought(DwT)框架,通过认知基础的思维链推理,引导多模态大模型将图表重建为可编辑的mxGraph XML代码。DwT将任务分为两阶段:粗粒度规划(处理感知结构与语义定义),以及结构感知代码生成(结合格式引导优化)。为支持评估,我们发布Plot2XML基准,包含247个真实世界科学图表及标准XML标注。在8个MLLM上的实验表明,该方法实现高保真、语义一致且结构正确的重构,人类评估确认其在准确性和视觉美感上均有显著优势,为静态图像转可执行表示提供了可扩展方案,推动机器对科学图形的理解。
原文摘要 · Abstract (English)
Scientific diagrams are vital tools for communicating structured knowledge across disciplines. However, they are often published as static raster images, losing symbolic semantics and limiting reuse. While Multimodal Large Language Models (MLLMs) offer a pathway to bridging vision and structure, existing methods lack semantic control and structural interpretability, especially on complex diagrams. We propose Draw with Thought (DwT), a training-free framework that guides MLLMs to reconstruct diagrams into editable mxGraph XML code through cognitively-grounded Chain-of-Thought reasoning. DwT enables interpretable and controllable outputs without model fine-tuning by dividing the task into two stages: Coarse-to-Fine Planning, which handles perceptual structuring and semantic specification, and Structure-Aware Code Generation, enhanced by format-guided refinement. To support evaluation, we release Plot2XML, a benchmark of 247 real-world scientific diagrams with gold-standard XML annotations. Extensive experiments across eight MLLMs show that our approach yields high-fidelity, semantically aligned, and structurally valid reconstructions, with human evaluations confirming strong alignment in both accuracy and visual aesthetics, offering a scalable solution for converting static visuals into executable representations and advancing machine understanding of scientific graphics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。