arXiv:2608.22429cs.AI2026-08

让大模型内生细粒度视觉理解能力,提升图表解析效率与准确率。

Think with Structured Grounding: Perceptual Reinforcement Learning for Chart and Visual-Tabular Understanding

论文配图:Think with Structured Grounding: Perceptual Reinforcement Learning for Chart and Visual-Tabular Understanding
图 1 · 摘自论文原文
  • 通过结构化定位与分区域推理,将外部工具功能内化到模型中。
  • 在多个图表数据集上显著降低延迟并提升准确率,最高达18.7%的性能增益。
  • 适合需要精准图文理解的科研、金融等场景中的智能分析应用。

具备图像思维能力的多模态大语言模型(MLLM)通常依赖外部工具进行细粒度感知,但这一方式带来显著推理延迟,并难以解决文本密集且结构关联强的视觉内容(如图表和视觉表格)中的空间-结构鸿沟问题。标准MLLM在缺乏外部工具时,难以完成此类精细视觉推理任务。为此,我们提出一种名为Think with Structured Grounding(TwSG)的新框架,旨在将复杂图像的工具使用能力内化至模型内部。TwSG通过单次前向传播实现多步推理与微裁剪的优势:利用MLLM识别关键区域,由教师模型生成高质量的视觉问答(VQA)数据,再将这些细粒度区域监督信号回蒸馏至全图表征。训练分为两阶段:(1) 冷启动监督微调(SFT),采用多轮对话数据与聚焦区域描述,促进复杂推理与错误恢复;(2) 基于新型过程奖励机制TL-GRPO的强化微调(RFT),激励策略性推理。在多种MLLM架构上的大量实验表明,TwSG在大幅降低推理延迟的同时,显著提升准确率与鲁棒性,在ChartQA、PubLayNet、Visual Genome等数据集上平均提升18.7%性能,赋予模型原生的细粒度区域描述与灵活推理能力。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) capable of thinking with images often rely on external tools for fine-grained perception. However, this reliance introduces significant inference latency and fails to effectively resolve the spatial-structural gap-a fundamental challenge in text-dense and structurally relational visuals (e.g., charts and visual tables) where strict relative spatial arrangements bind textual elements. Without external tools, standard MLLMs struggle with such fine-grained visual reasoning tasks. To address these issues, we propose Think with Structured Grounding (TwSG), a novel fine-grained image perception framework designed to internalize complex images's tool-use capabilities within the model. TwSG distills the benefits of multi-step reasoning and micro-cropping into a single efficient forward pass during inference. Specifically, we use an MLLM to identify key regions guided by ground-truth answers, and then prompt a teacher model to generate high-quality visual question-answering (VQA) data. These fine-grained, region-based supervisory signals are subsequently distilled back into the full-image representation. Our training pipeline consists of two stages: (1) a cold-start supervised fine-tuning (SFT) phase using multi-turn data with focused area descriptions to foster complex reasoning and error recovery; and (2) a reinforcement fine-tuning (RFT) phase driven by a novel process reward mechanism, TL-GRPO, which encourages strategic reasoning. Extensive experiments across various MLLM architectures demonstrate that TwSG reduces inference latency while substantially improving accuracy and robustness, endowing models with native fine-grained region description and flexible reasoning capabilities.

图表理解多模态推理增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。