将视觉解析与语言推理分离,提升数学图像题解题效率
Integrating Visual Interpretation and Linguistic Reasoning for Math Problem Solving
- 用专用模型把图片转成文字描述,再交由大模型推理
- 在几何类数学题上性能超越现有端到端模型
- 适合需要快速适配新大模型的多模态应用
当前大型视觉语言模型(LVLMs)通常通过连接模块将视觉特征与语言模型文本嵌入对齐,并采用端到端训练实现统一的多模态理解。有效对齐依赖高质量预训练数据和精心设计的训练流程。现有LVLMs在处理复杂视觉语言推理任务时表现不足,其推理能力明显落后于纯语言模型。本文提出范式转变:不训练端到端的视觉语言推理模型,而是基于现有的视觉解析专家和文本推理大模型,构建解耦推理框架。该方法利用(1)专用视觉语言模型将图像内容转化为文本描述,(2)大语言模型根据生成的文本和原始问题进行推理。此策略通过优化已有模型协同工作,实现低成本的多模态模型开发,避免从零开始训练视觉语言模型。通过将图像转化为语言模型兼容的文本表示,便于未来低成本、灵活地适配更强大的新语言模型。我们引入结果奖励联合调优策略,优化视觉解析与语言推理模型间的协作。在多个视觉语言基准上的评估表明,该解耦框架优于近期主流LVLMs,尤其在视觉密集型几何数学问题上表现显著提升。代码已开源:https://github.com/guozix/DVLR。
原文摘要 · Abstract (English)
Current large vision-language models (LVLMs) typically employ a connector module to link visual features with text embeddings of large language models (LLMs) and use end-to-end training to achieve multi-modal understanding in a unified process. Effective alignment needs high-quality pre-training data and a carefully designed training process. Current LVLMs face challenges when addressing complex vision-language reasoning tasks, with their reasoning capabilities notably lagging behind those of LLMs. This paper proposes a paradigm shift: instead of training end-to-end vision-language reasoning models, we advocate for developing a decoupled reasoning framework based on existing visual interpretation specialists and text-based reasoning LLMs. Our approach leverages (1) a dedicated vision-language model to transform the visual content of images into textual descriptions and (2) an LLM to perform reasoning according to the visual-derived text and the original question. This method presents a cost-efficient solution for multi-modal model development by optimizing existing models to work collaboratively, avoiding end-to-end development of vision-language models from scratch. By transforming images into language model-compatible text representations, it facilitates future low-cost and flexible upgrades to upcoming powerful LLMs. We introduce an outcome-rewarded joint-tuning strategy to optimize the cooperation between the visual interpretation and linguistic reasoning model. Evaluation results on vision-language benchmarks demonstrate that the decoupled reasoning framework outperforms recent LVLMs. Our approach yields particularly significant performance gains on visually intensive geometric mathematics problems. The code is available: https://github.com/guozix/DVLR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。