用TikZ代码提升大模型对几何结构的理解与推理能力
GeoTikzBridge: Advancing Multimodal Code Generation for Geometric Perception and Reasoning
- 通过生成TikZ代码增强局部几何感知能力
- 在250万图像-代码对上训练，性能超越现有开源模型
- 可作为模块嵌入任意大模型，提升几何推理效果
多模态大语言模型虽具强大感知与推理能力，但在细粒度几何结构识别方面表现不足。为此，我们提出GeoTikzBridge框架，通过TikZ代码生成强化局部几何感知与视觉推理。该框架构建两个模型：基于250万对图像-TikZ数据的GeoTikzBridge-Base模型（较现有开源数据集大16倍），采用迭代数据扩展与局部几何变换策略训练；以及首个支持视觉推理的指令增强型TikZ数据集GeoTikz-Instruct上微调的GeoTikzBridge-Instruct模型。大量实验表明，本模型在开源多模态大模型中达到领先水平。此外，其可作为即插即用的推理模块，显著提升任意大模型在几何问题求解中的表现。数据集与代码已公开于https://github.com/sjy-1995/GeoTikzBridge。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have recently demonstrated remarkable perceptual and reasoning abilities. However, they struggle to perceive fine-grained geometric structures, constraining their ability of geometric understanding and visual reasoning. To address this, we propose GeoTikzBridge, a framework that enhances local geometric perception and visual reasoning through tikz-based code generation. Within this framework, we build two models supported by two complementary datasets. The GeoTikzBridge-Base model is trained on GeoTikz-Base dataset, the largest image-to-tikz dataset to date with 2.5M pairs (16 $\times$ larger than existing open-sourced datasets). This process is achieved via iterative data expansion and a localized geometric transformation strategy. Subsequently, GeoTikzBridge-Instruct is fine-tuned on GeoTikz-Instruct dataset which is the first instruction-augmented tikz dataset supporting visual reasoning. Extensive experimental results demonstrate that our models achieve state-of-the-art performance among open-sourced MLLMs. Furthermore, GeoTikzBridge models can serve as plug-and-play reasoning modules for any MLLM(LLM), enhancing reasoning performance in geometric problem-solving. Datasets and codes are publicly available at: https://github.com/sjy-1995/GeoTikzBridge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。