arXiv:2509.09286cs.CV2025-09被引 3

让AI学会根据图表特点自选推理方式,更准更可信。

Visual Programmability: A Guide for Code-as-Thought in Chart Understanding

  • 用代码符号化表达图表信息,可验证且灵活适应复杂场景。
  • 在多个图表理解基准上表现优异,准确率显著优于单一策略模型。
  • 适合需要动态决策的视觉推理任务,如智能报告生成、数据分析助手。

图表理解是检验视觉-语言模型推理能力的关键挑战。现有方法存在明显局限:部分依赖外部工具,易受预设工具集限制;另一些则微调专用模型,通常采用单一推理策略(如基于文本的链式思维)。文本推理中间步骤难以验证,阻碍了基于事实准确性的强化学习信号应用。为此,我们提出代码即思维(Code-as-Thought, CaT)方法,将图表视觉信息以可验证的符号格式表示。关键发现是该策略必须具备自适应性:固定代码实现对复杂图表效果不佳。由此提出视觉可编程性——一种可学习属性,用于判断图表-问题对更适合代码推理还是直接视觉分析。我们在自适应框架中实现此概念,让模型通过强化学习选择路径:或走CaT路径,或走直接视觉推理路径。训练采用新颖的双奖励系统,结合数据准确性奖励(防止数值幻觉)和决策奖励(教会模型何时使用何种策略),避免模型固化于单一模式。实验表明,在多种图表理解基准上均表现出强而稳健的性能。本工作证明,视觉-语言模型不仅能推理,还能学会如何推理,动态选择最优路径。

原文摘要 · Abstract (English)

Chart understanding presents a critical test to the reasoning capabilities of Vision-Language Models (VLMs). Prior approaches face critical limitations: some rely on external tools, making them brittle and constrained by a predefined toolkit, while others fine-tune specialist models that often adopt a single reasoning strategy, such as text-based chain-of-thought (CoT). The intermediate steps of text-based reasoning are difficult to verify, which complicates the use of reinforcement-learning signals that reward factual accuracy. To address this, we propose a Code-as-Thought (CaT) approach to represent the visual information of a chart in a verifiable, symbolic format. Our key insight is that this strategy must be adaptive: a fixed, code-only implementation consistently fails on complex charts where symbolic representation is unsuitable. This finding leads us to introduce Visual Programmability: a learnable property that determines if a chart-question pair is better solved with code or direct visual analysis. We implement this concept in an adaptive framework where a VLM learns to choose between the CaT pathway and a direct visual reasoning pathway. The selection policy of the model is trained with reinforcement learning using a novel dual-reward system. This system combines a data-accuracy reward to ground the model in facts and prevent numerical hallucination, with a decision reward that teaches the model when to use each strategy, preventing it from defaulting to a single reasoning mode. Experiments demonstrate strong and robust performance across diverse chart-understanding benchmarks. Our work shows that VLMs can be taught not only to reason but also how to reason, dynamically selecting the optimal reasoning pathway for each task.

视觉推理代码思维自适应决策

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。