用代码自动生成图表数据，让视觉语言模型自己提升图表理解能力
Chart-CoCa: Self-Improving Chart Understanding of Vision LMs via Code-Driven Synthesis and Candidate-Conditioned Answering
- 通过代码生成与执行，自动构建图文问答三元组数据
- 在无标注数据下实现最高15.50分的准确率提升
- 适合想改进图表理解的AI研究者和开发者
视觉语言模型（VLMs）在图表理解任务中常表现不佳，尤其在精准描述与复杂推理方面。合成数据生成是潜在解决方案，但通常面临标签噪声问题。为此，我们提出一个图表合成管道，通过代码生成与执行自动生成对齐的图表-问题-答案三元组，确保合成数据可靠性且无需人工干预。此外，受测试时扩展启发，我们设计了候选条件化回答机制：VLM对每个查询生成多个回答，再通过上下文整合形成最终答案。实验表明，在完全自提升范式下，无需人工标注数据或外部模型，准确率最高提升15.50分。
原文摘要 · Abstract (English)
Vision Language Models (VLMs) often struggle with chart understanding tasks, particularly in accurate chart description and complex reasoning. Synthetic data generation is a promising solution, while usually facing the challenge of noise labels. To address this challenge, we first introduce a chart synthesis pipeline that generates aligned chart-question-answer triplets through code generation and execution, ensuring the reliability of synthetic data without human intervention. Furthermore, inspired by test-time scaling that increases inference budget and thereby improves performance, we design a candidate-conditioned answering process. The VLM first generates multiple responses per query, and then synthesizes the final answer by contextualizing these candidates. Experiments demonstrate significant improvements, with up to 15.50 points accuracy gain over the initial VLM, in a fully self-improving paradigm without either human-labeled data or external models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。