用合成数据提升视觉语言模型在图表理解等任务上的表现。
MM-GEN: Enhancing Task Performance Through Targeted Multimodal Data Curation
- 用强模型生成特定任务的高质量合成文本，分三步精准构建数据。
- 使Llava-1.5在空间推理上提升29%，图表理解提升15%。
- 比人工标注数据效果更好，适合需要高精度图文理解的场景。
视觉语言模型（VLMs）虽高效，但在专业任务上表现不足，如Llava-1.5在图表理解方面因缺乏特定数据而表现不佳。现有训练数据来自通用数据集，难以捕捉任务细节。我们提出MM-Gen，一种可扩展的方法，通过更强模型为候选图像生成任务特定的高质量合成文本。该方法采用三阶段流程：将数据分组、根据任务描述生成目标文本、过滤冗余与异常数据。用MM-Gen生成的数据微调VLMs后，性能显著提升，使Llava-1.5（7B）在空间推理上提高29%，图表理解提升15%。相比人工标注的描述数据，MM-Gen带来的提升最高达1.6倍，验证了其在增强特定任务性能方面的有效性，弥合通用数据集与专业化需求之间的差距。代码已开源。
原文摘要 · Abstract (English)
Vision-language models (VLMs) are highly effective but often underperform on specialized tasks; for example, Llava-1.5 struggles with chart and diagram understanding due to scarce task-specific training data. Existing training data, sourced from general-purpose datasets, fails to capture the nuanced details needed for these tasks. We introduce MM-Gen, a scalable method that generates task-specific, high-quality synthetic text for candidate images by leveraging stronger models. MM-Gen employs a three-stage targeted process: partitioning data into subgroups, generating targeted text based on task descriptions, and filtering out redundant and outlier data. Fine-tuning VLMs with data generated by MM-Gen leads to significant performance gains, including 29% on spatial reasoning and 15% on diagram understanding for Llava-1.5 (7B). Compared to human-curated caption data, MM-Gen achieves up to 1.6x better improvements for the original models, proving its effectiveness in enhancing task-specific VLM performance and bridging the gap between general-purpose datasets and specialized requirements. Code available at https://github.com/sjoshi804/MM-Gen.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。