构建首个自验证的科学图表问答数据集,提升开源模型表现
VeriSciQA: An Auto-Verified Dataset for Scientific Visual Question Answering
- 用图文对齐文本生成问题答案,再通过图表验证真伪
- 数据集含20272组问答,覆盖20个领域和12种图类型
- 经实测,该数据集可显著提升开源模型性能
大型视觉语言模型在科学应用中前景广阔,但开源模型在科学图表问答(SVQA)任务上仍表现不佳。主要瓶颈在于缺乏公开、大规模、高质量的SVQA数据集。尽管近期研究利用视觉语言模型大规模合成数据,但其生成的问答对存在系统性错误,源于模型固有局限及图文信息不对称。为此,我们提出一种跨模态验证框架:仅基于引用图表的段落生成问题与答案,再通过实际图表进行验证,利用科学论文中固有的图文对应关系过滤错误配对。基于此框架构建了VeriSciQA数据集,包含20,272组问答对,覆盖20个科学领域和12种图类型。难度评估显示,最佳开源模型准确率为65%,而最佳专有模型达80.5%,表明仍有提升空间。在该数据集上微调的模型在多个SVQA基准上持续取得进步,性能随数据量增长而提升,优于现有数据集训练模型。人工评估进一步验证了数据质量提升。结果表明,通过该可扩展框架持续扩充数据,可进一步推动开源社区的SVQA能力。数据集已公开于https://huggingface.co/datasets/datajuicer/VeriSciQA。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) show promise for scientific applications, yet open-source models still struggle with Scientific Visual Question Answering (SVQA), namely answering questions about figures from scientific papers. A key bottleneck is the lack of public, large-scale, high-quality SVQA datasets. Although recent work uses LVLMs to synthesize data at scale, we identify systematic errors in their resulting QA pairs, stemming from LVLMs' inherent limitations and information asymmetry between figures and text. To address these challenges, we propose a Cross-Modal verification framework that generates questions and answers purely from figure-citing paragraphs, then verifies them against the figures themselves, leveraging the inherent text-figure alignment in scientific papers to filter out erroneous QA pairs. We instantiate this framework to curate VeriSciQA, a dataset of 20,272 QA pairs spanning 20 scientific domains and 12 figure types. Difficulty assessment reveals a notable accuracy gap between the best open-source model (65%) and the best proprietary model (80.5%), demonstrating room for improvement. Moreover, models fine-tuned on VeriSciQA achieve consistent improvements on SVQA benchmarks, with performance gains that scale with data size, surpassing models trained on existing datasets. Human evaluation further validates the improved quality of VeriSciQA. These results demonstrate that continued data expansion via our scalable framework can further advance SVQA capability in the open-source community. Our dataset is publicly available at https://huggingface.co/datasets/datajuicer/VeriSciQA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。