让AI看图推理更准：通过视觉定位纠正图表误解
ChartPoint: Guiding MLLMs with Grounding Reflection for Chart Reasoning
- 引入反思式思维链，在推理中加入视觉位置标注
- 构建6.2万样本数据集，含逐步推理与重绘图像
- 模型在图表基准测试中提升5.04%，适合图表分析场景
多模态大语言模型（MLLMs）已成为图表理解的强大工具，但其严重依赖OCR提取内容，当图表文字信息稀疏时易产生数值幻觉。现有方法聚焦于扩展指令规模，却未解决根本问题——基于视觉感知的推理能力不足。本文发现，MLLMs在图表元素和比例关系上的视觉定位能力薄弱，推理结果无法对应实际位置。为此，我们提出PointCoT，将反思性交互融入图表的思维链推理中。通过提示模型生成边界框并基于位置标注重绘图表，建立文本推理步骤与视觉定位区域之间的联系。我们进一步设计自动化流水线，构建ChartPoint-SFT-62k数据集，包含19.2K高质量图表样本，带有逐步思维链、边界框及重绘可视化。基于此数据，我们训练出两个指令微调模型ChartPointQ2和ChartPointQ2.5，显著优于当前最佳模型，在ChartBench上提升5.04%。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have emerged as powerful tools for chart comprehension. However, they heavily rely on extracted content via OCR, which leads to numerical hallucinations when chart textual annotations are sparse. While existing methods focus on scaling instructions, they fail to address the fundamental challenge, i.e., reasoning with visual perception. In this paper, we identify a critical observation: MLLMs exhibit weak grounding in chart elements and proportional relationships, as evidenced by their inability to localize key positions to match their reasoning. To bridge this gap, we propose PointCoT, which integrates reflective interaction into chain-of-thought reasoning in charts. By prompting MLLMs to generate bounding boxes and re-render charts based on location annotations, we establish connections between textual reasoning steps and visual grounding regions. We further introduce an automated pipeline to construct ChartPoint-SFT-62k, a dataset featuring 19.2K high-quality chart samples with step-by-step CoT, bounding box, and re-rendered visualizations. Leveraging this data, we develop two instruction-tuned models, ChartPointQ2 and ChartPointQ2.5, which outperform state-of-the-art across several chart benchmarks, e.g., +5.04\% on ChartBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。