提升视觉语言模型的图形算术能力，让其更懂图表与几何
Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understanding
- 基于皮亚杰认知理论设计新训练方法，增强对视觉变换不变性的理解
- 在多个图表数据集上平均提升4.6%，仅需60%训练数据即达良好效果
- 适合希望改进模型视觉推理能力的研究者与应用开发者
视觉语言模型在多模态任务中取得显著进展，但在物体计数、长度比较等基础视觉算术任务上表现不佳，而这些能力对图表理解与几何推理至关重要。本文通过一系列探针任务分析其根本原因，发现预训练视觉编码器已具备足够信息，问题出在文本解码器无法正确解析用于算术推理的信息。为此，我们提出CogAlign，一种受皮亚杰认知发展理论启发的后训练策略，训练模型识别视觉变换下的不变属性。实验证明，该方法显著提升三种不同VLM在探针任务上的表现，并在CHOCOLATE和MATH-VISION数据集上分别平均提升4.6%和2.9%，优于或媲美监督微调，且仅需60%训练数据。结果表明CogAlign有效提升基础视觉算术能力及其在下游任务中的迁移性。
原文摘要 · Abstract (English)
Vision Language Models (VLMs) have achieved remarkable progress in multimodal tasks, yet they often struggle with visual arithmetic, seemingly simple capabilities like object counting or length comparison, which are essential for relevant complex tasks like chart understanding and geometric reasoning. In this work, we first investigate the root causes of this deficiency through a suite of probing tasks focusing on basic visual arithmetic. Our analysis reveals that while pre-trained vision encoders typically capture sufficient information, the text decoder often fails to decode it correctly for arithmetic reasoning. To address this, we propose CogAlign, a novel post-training strategy inspired by Piaget's theory of cognitive development. CogAlign trains VLMs to recognize invariant properties under visual transformations. We demonstrate that this approach significantly improves the performance of three diverse VLMs on our proposed probing tasks. Furthermore, CogAlign enhances performance by an average of 4.6% on CHOCOLATE and 2.9% on MATH-VISION, outperforming or matching supervised fine-tuning methods while requiring only 60% less training data. These results highlight the effectiveness and generalizability of CogAlign in improving fundamental visual arithmetic capabilities and their transfer to downstream tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。