让AI像人一样画辅助线、做变换解几何题。
GeoSketch: A Neural-Symbolic Approach to Geometric Multimodal Reasoning with Auxiliary Line Construction and Affine Transformation
- 用神经符号框架实现画线、变换等动态操作
- 在390道题上准确率显著高于静态图像方法
- 适合需要空间推理的数学竞赛或教育AI
几何问题求解对多模态大模型构成独特挑战,要求同时理解文本与图形,并进行迭代式空间推理。现有方法将图形视为静态图像,缺乏动态操作能力——而人类解题常依赖辅助线构造与仿射变换。我们提出GeoSketch,一种神经符号框架,将几何推理重构为感知-推理-动作的闭环交互过程。该框架包含:(1) 将图形抽象为结构化逻辑形式的感知模块,(2) 基于几何定理决定下一步推理的符号推理模块,(3) 执行画辅助线或应用变换等操作并更新图形的绘图动作模块。通过两阶段训练流程:在2,000条人工标注轨迹上进行监督微调,再使用密集符号奖励进行强化学习以提升鲁棒性与策略探索能力。为评估该范式,我们构建了包含390道需辅助构造或仿射变换的高质量几何题基准集。实验表明,相比强基线模型,GeoSketch在步骤级推理准确率和问题求解成功率上均有显著提升。通过整合层次化决策、可执行视觉动作与符号验证,该框架推动多模态推理从静态理解迈向动态、可验证的交互,为解决复杂空间问题奠定新基础。
原文摘要 · Abstract (English)
Geometric Problem Solving (GPS) poses a unique challenge for Multimodal Large Language Models (MLLMs), requiring not only the joint interpretation of text and diagrams but also iterative visuospatial reasoning. While existing approaches process diagrams as static images, they lack the capacity for dynamic manipulation - a core aspect of human geometric reasoning involving auxiliary line construction and affine transformations. We present GeoSketch, a neural-symbolic framework that recasts geometric reasoning as an interactive perception-reasoning-action loop. GeoSketch integrates: (1) a Perception module that abstracts diagrams into structured logic forms, (2) a Symbolic Reasoning module that applies geometric theorems to decide the next deductive step, and (3) a Sketch Action module that executes operations such as drawing auxiliary lines or applying transformations, thereby updating the diagram in a closed loop. To train this agent, we develop a two-stage pipeline: supervised fine-tuning on 2,000 symbolic-curated trajectories followed by reinforcement learning with dense, symbolic rewards to enhance robustness and strategic exploration. To evaluate this paradigm, we introduce the GeoSketch Benchmark, a high-quality set of 390 geometry problems requiring auxiliary construction or affine transformations. Experiments on strong MLLM baselines demonstrate that GeoSketch significantly improves stepwise reasoning accuracy and problem-solving success over static perception methods. By unifying hierarchical decision-making, executable visual actions, and symbolic verification, GeoSketch advances multimodal reasoning from static interpretation to dynamic, verifiable interaction, establishing a new foundation for solving complex visuospatial problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。