用函数链生成精准多样的图表推理数据,提升大模型理解能力。
Chain of Functions: A Programmatic Pipeline for Fine-Grained Chart Reasoning Data
- 通过原子函数组合生成推理路径,保证数据精度与多样性。
- 构建1.4k精细分析用数据集,50k用于模型训练,效果领先。
- 适合研究多模态模型推理、数据生成与可解释性方向的人看。
视觉推理对多模态大语言模型(MLLMs)解决复杂图表问题至关重要,但高质量推理数据仍稀缺。现有方法依赖(M)LLMs直接生成,常导致精度和多样性不足。本文提出 extit{Chain of Functions (CoF)},一种程序化推理数据生成流水线,利用自由探索的原子函数(如最大值、算术运算)组合生成多样函数链,再通过中等规模开源LLM转化为语言推理与问题。该方法优势显著:1)精度高,函数约束减少幻觉;2)多样性强,函数链枚举支持多样化问题分类;3)可解释性强,函数链即内置推理过程,支持细粒度评估;4)实用性高,无需超大规模模型。基于CoF构建了 extit{ChartCoF}数据集,含1.4k复杂推理问答用于细粒度分析,50k用于推理增强。细粒度评估揭示不同MLLM在各类问题上的表现差异,微调实验表明其在同类规模模型中达到最优性能。该函数约束生成范式亦可推广至其他场景。
原文摘要 · Abstract (English)
Visual reasoning is crucial for multimodal large language models (MLLMs) to address complex chart queries, yet high-quality rationale data remains scarce. Existing methods leveraged (M)LLMs for data generation, but direct prompting often yields limited precision and diversity. In this paper, we propose \textit{Chain of Functions (CoF)}, a novel programmatic reasoning data generation pipeline that utilizes freely-explored reasoning paths as supervision to ensure data precision and diversity. Specifically, it starts with human-free exploration among the atomic functions (e.g., maximum data and arithmetic operations) to generate diverse function chains, which are then translated into linguistic rationales and questions with only a moderate open-sourced LLM. \textit{CoF} provides multiple benefits: 1) Precision: function-governed generation reduces hallucinations compared to freeform generation; 2) Diversity: enumerating function chains enables varied question taxonomies; 3) Explainability: function chains serve as built-in rationales, allowing fine-grained evaluation beyond overall accuracy; 4) Practicality: eliminating reliance on extremely large models. Employing \textit{CoF}, we construct the \textit{ChartCoF} dataset, with 1.4k complex reasoning Q\&A for fine-grained analysis and 50k Q\&A for reasoning enhancement. The fine-grained evaluation on \textit{ChartCoF} reveals varying performance across question taxonomies for each MLLM, and the experiments also show that finetuning with \textit{ChartCoF} achieves state-of-the-art performance among same-scale MLLMs on widely used benchmarks. Furthermore, the novel paradigm of function-governed rationale generation in \textit{CoF} could inspire broader applications beyond charts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。