arXiv:2510.15040cs.CVcs.CL2025-10被引 1

用少量问题生成海量训练数据,提升模型对图表的推理能力

Composition-Grounded Data Synthesis for Visual Reasoning

  • 将问题拆解为感知与推理基础单元,重组生成新问答对
  • 在未见过的问题上推理性能显著提升,尤其擅长复杂组合题
  • 适用于图表、网页等多类视觉场景,可避免过拟合

预训练的多模态大语言模型在多种多模态任务中表现优异,但在标注困难的人工图像领域(如图表、渲染文档、网页)仍受限于缺乏大规模人工标注的推理数据。本文提出COGS(Composition-Grounded data Synthesis)框架,仅需少量种子问题即可高效生成大量合成问答对,赋予多模态大模型高级推理能力。核心思路是将每个种子问题分解为基本感知与推理因子,再通过系统化重组生成新图像下的问答对,并配以子问题和中间答案,支持基于因子级别的过程奖励强化学习。在图表推理任务上,使用该方法训练的模型在未见问题上表现显著提升,尤其在重推理与组合型问题上收益最大。此外,混合不同种子数据进行因子级训练能更好跨数据集迁移,表明其具备泛化能力而非特定数据集过拟合。实验还验证了该框架可扩展至网页等其他视觉领域。

原文摘要 · Abstract (English)

Pretrained multi-modal large language models (MLLMs) demonstrate strong performance on diverse multimodal tasks, but remain limited in reasoning capabilities for domains where annotations are difficult to collect. In this work, we focus on artificial image domains such as charts, rendered documents, and webpages, which are abundant in practice yet lack large-scale human annotated reasoning datasets. We introduce COGS (COmposition-Grounded data Synthesis), a data-efficient framework for equipping MLLMs with advanced reasoning abilities from a small set of seed questions. The key idea is to decompose each seed question into primitive perception and reasoning factors, which can then be systematically recomposed with new images to generate large collections of synthetic question-answer pairs. Each generated question is paired with subquestions and intermediate answers, enabling reinforcement learning with factor-level process rewards. Experiments on chart reasoning show that COGS substantially improves performance on unseen questions, with the largest gains on reasoning-heavy and compositional questions. Moreover, training with a factor-level mixture of different seed data yields better transfer across multiple datasets, suggesting that COGS induces generalizable capabilities rather than dataset-specific overfitting. We further demonstrate that the framework extends beyond charts to other domains such as webpages.

多模态推理增强数据合成图表理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。