arXiv:2501.10768cs.AI2025-01ICLR被引 14

让大模型读懂物理图示并推理电路问题,准确率显著提升。

MAPS: Advancing Multi-Modal Reasoning in Expert-Level Physical Science

  • 用合成数据微调视觉语言模型,专门理解复杂物理图示。
  • 在大学电路题上准确率超越现有模型,达82.3%。
  • 适合需要物理推理的科研与教育场景使用。

当前多模态大语言模型(MLLM)虽在通用视觉推理中表现优异,但在需理解复杂物理结构和多模态定量分析的物理领域仍显不足。为此,我们提出基于MLLM的新框架MAPS(Multi-Modal Scientific Reasoning with Physics Perception and Simulation),将专家级多模态推理任务分解为:通过物理感知模型(PPM)理解物理图示,以及利用模拟器结合物理知识进行推理。PPM通过在精心设计的合成数据上微调视觉语言模型获得,该数据包含配对的物理图示与仿真语言描述。推理阶段,MAPS融合PPM生成的图示仿真描述及链式模拟过程结果,由MLLM推导出逻辑依据与最终答案。在自建的大学级电路分析题集上验证,MAPS显著提升MLLM推理准确率,达到82.3%,优于所有现有模型。结果表明,MAPS为增强MLLM的多模态科学推理能力提供了可行路径。代码、模型与数据集将在论文发表后公开。

原文摘要 · Abstract (English)

Pre-trained on extensive text and image corpora, current Multi-Modal Large Language Models (MLLM) have shown strong capabilities in general visual reasoning tasks. However, their performance is still lacking in physical domains that require understanding diagrams with complex physical structures and quantitative analysis based on multi-modal information. To address this, we develop a new framework, named Multi-Modal Scientific Reasoning with Physics Perception and Simulation (MAPS) based on an MLLM. MAPS decomposes expert-level multi-modal reasoning task into physical diagram understanding via a Physical Perception Model (PPM) and reasoning with physical knowledge via a simulator. The PPM module is obtained by fine-tuning a visual language model using carefully designed synthetic data with paired physical diagrams and corresponding simulation language descriptions. At the inference stage, MAPS integrates the simulation language description of the input diagram provided by PPM and results obtained through a Chain-of-Simulation process with MLLM to derive the underlying rationale and the final answer. Validated using our collected college-level circuit analysis problems, MAPS significantly improves reasoning accuracy of MLLM and outperforms all existing models. The results confirm MAPS offers a promising direction for enhancing multi-modal scientific reasoning ability of MLLMs. We will release our code, model and dataset used for our experiments upon publishing of this paper.

多模态物理推理大模型电路分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。