用代码形式生成高质量多模态数据，提升视觉语言模型性能
World to Code: Multi-modal Data Generation via Self-Instructed Compositional Captioning and Filtering
- 用自指导组合描述和过滤策略生成多模态数据
- 在多个视觉问答与视觉定位任务上显著提升模型表现
- 适合研究多模态数据生成与视觉语言对齐的学者
近期视觉语言模型（VLMs）的发展以及高质量多模态对齐数据的稀缺，推动了合成多模态数据生成的研究。传统方法依赖于图文描述与OCR混合，或使用强大的VLM API及昂贵的人工标注。本文提出World to Code（W2C），一个精心设计的多模态数据构建流程，将最终生成结果组织为Python代码格式。该流程利用VLM自身通过不同提示提取跨模态信息，并采用一致性过滤策略进一步筛选输出。实验表明，W2C在多个现有视觉问答和视觉定位基准上均提升了不同VLM的表现。进一步分析显示，VLM的新代码解析能力在跨模态等价性上优于常用的细节描述能力。代码已开源：https://github.com/foundation-multimodal-models/World2Code。
原文摘要 · Abstract (English)
Recent advances in Vision-Language Models (VLMs) and the scarcity of high-quality multi-modal alignment data have inspired numerous researches on synthetic VLM data generation. The conventional norm in VLM data construction uses a mixture of specialists in caption and OCR, or stronger VLM APIs and expensive human annotation. In this paper, we present World to Code (W2C), a meticulously curated multi-modal data construction pipeline that organizes the final generation output into a Python code format. The pipeline leverages the VLM itself to extract cross-modal information via different prompts and filter the generated outputs again via a consistency filtering strategy. Experiments have demonstrated the high quality of W2C by improving various existing visual question answering and visual grounding benchmarks across different VLMs. Further analysis also demonstrates that the new code parsing ability of VLMs presents better cross-modal equivalence than the commonly used detail caption ability. Our code is available at https://github.com/foundation-multimodal-models/World2Code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。