arXiv:2410.18798cs.CL2024-10EMNLP被引 34

用代码做桥梁,把大模型的图表推理能力迁移到多模态模型。

Distill Visual Chart Reasoning Ability from LLMs to MLLMs

  • 用代码作为中介,将图表转为文本,实现跨模态理解
  • 生成3000张高质图表与2万组问答对,提升模型推理能力
  • 适合需要增强图表理解与通用推理能力的研究者

解决复杂的图表问答任务需要多模态大语言模型具备高级视觉推理能力,包括从视觉输入中识别关键信息并进行推理。尽管对多模态模型进行推理微调至关重要,但收集和标注图表及问题成本高、难扩展,且常导致低质量标注。为此,我们提出代码作为中间表示的翻译方法(CIT),一种低成本、高效且可扩展的数据合成方法,用于将大语言模型的视觉推理能力蒸馏到多模态模型中。代码作为中介,将图表的视觉表示转换为文本表示,使语言模型能够理解跨模态信息并生成相应的推理链。通过这种方法,可利用基于文本的生成技术扩展图表绘制代码,生成高质量的问答对以训练模型。该方法构建了ReachQA数据集,包含3000个推理密集型图表和20000组问答对,显著提升多模态模型的识别与推理能力。实验表明,使用ReachQA微调的模型不仅在图表相关任务上表现优异,还在通用推理基准上取得性能提升。代码与数据集已公开于https://github.com/hewei2001/ReachQA。

原文摘要 · Abstract (English)

Solving complex chart Q&A tasks requires advanced visual reasoning abilities in multimodal large language models (MLLMs), including recognizing key information from visual inputs and conducting reasoning over it. While fine-tuning MLLMs for reasoning is critical, collecting and annotating charts and questions is expensive, hard to scale, and often results in low-quality annotations. To address this, we propose Code-as-Intermediary Translation (CIT), a cost-effective, efficient and scalable data synthesis method for distilling visual reasoning abilities from LLMs to MLLMs. The code serves as an intermediary that translates visual chart representations into textual representations, enabling language models to understand cross-modal information and generate reasoning chains accordingly. In this way, we can employ text-based synthesizing techniques to expand chart-plotting code and generate high-quality Q&A pairs for training models. This produces ReachQA, a dataset containing 3k reasoning-intensive charts and 20k Q&A pairs to enhance both recognition and reasoning abilities of MLLMs. Experiments show that models fine-tuned with ReachQA not only perform well on chart-related tasks but also show performance gains on general reasoning benchmarks. The code and dataset are publicly available at https://github.com/hewei2001/ReachQA.

图表理解知识蒸馏多模态数据合成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。