用模块化代码让视觉语言模型精准解几何题
GeoCoder: Solving Geometry Problems by Generating Modular Code through Vision-Language Models
- 把几何公式编成可调用的代码库，让模型执行计算
- 在GeomVerse上平均准确率提升16%以上
- 适合需要严谨数学推理的AI研究者
几何问题求解需要处理多模态输入并有效运用数学知识。尽管视觉语言模型在多模态任务中取得进展，但在几何问题上仍表现不佳，主要受限于无法进行预训练中未见的数学运算（如计算任意角的余弦值）以及难以正确应用几何公式。为此，我们提出GeoCoder，通过模块化代码微调生成并执行代码，利用预定义的几何函数库完成计算。代码执行实现精确、确定性结果，克服了自回归生成的随机性；函数库则减少公式使用错误。我们还提出RAG-GeoCoder，引入非参数记忆模块从函数库中检索公式，降低对参数记忆的依赖。模块化代码微调显著提升视觉语言模型的几何推理能力，在GeomVerse数据集上，针对不同复杂度的问题，平均准确率相较其他微调方法提升超过16%。
原文摘要 · Abstract (English)
Geometry problem-solving demands advanced reasoning abilities to process multimodal inputs and employ mathematical knowledge effectively. Vision-language models (VLMs) have made significant progress in various multimodal tasks. Yet, they still struggle with geometry problems and are significantly limited by their inability to perform mathematical operations not seen during pre-training, such as calculating the cosine of an arbitrary angle, and by difficulties in correctly applying relevant geometry formulas. To overcome these challenges, we present GeoCoder, which leverages modular code-finetuning to generate and execute code using a predefined geometry function library. By executing the code, we achieve accurate and deterministic calculations, contrasting the stochastic nature of autoregressive token prediction, while the function library minimizes errors in formula usage. We also propose a multimodal retrieval-augmented variant of GeoCoder, named RAG-GeoCoder, which incorporates a non-parametric memory module for retrieving functions from the geometry library, thereby reducing reliance on parametric memory. Our modular code-finetuning approach enhances the geometric reasoning capabilities of VLMs, yielding an average improvement of over 16% across various question complexities on the GeomVerse dataset compared to other finetuning methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。