仅用文本生成高质量视觉语言模型训练数据，降低成本。
Text-Only Data Synthesis for Vision Language Model Training
- 三阶段框架：从少量文本种子生成120万条多样描述
- 47.1万条指令数据支持复杂推理任务
- 无需真实图像，适合资源有限的研究者
训练视觉语言模型通常需要大规模高质量的图文对，但获取或合成这类数据成本高昂。相比之下，文本数据丰富且廉价，因此提出一个问题：能否仅通过文本合成高质量多模态训练数据？为此，我们提出一个跨融合的三阶段多模态数据合成框架，生成两个数据集：Unicorn-1.2M 和 Unicorn-471K-Instruction。第一阶段：多样化的标题生成，利用大语言模型（LLMs）扩展稀疏的标题种子，构建120万条语义多样、高质量的标题。第二阶段：指令微调数据生成，将其中47.1万条标题转化为多轮指令微调任务，支持复杂推理。第三阶段：模态表征迁移，将这些文本表征转换为视觉表征，生成多样化的合成图像表示。该三阶段流程实现了无需真实图像即可构建Unicorn-1.2M用于预训练和Unicorn-471K-Instruction用于指令微调。在不依赖真实图像的前提下，保持了数据质量和多样性，为视觉语言模型训练提供了低成本、可扩展的解决方案。
原文摘要 · Abstract (English)
Training vision-language models (VLMs) typically requires large-scale, high-quality image-text pairs, but collecting or synthesizing such data is costly. In contrast, text data is abundant and inexpensive, prompting the question: can high-quality multimodal training data be synthesized purely from text? To tackle this, we propose a cross-integrated three-stage multimodal data synthesis framework, which generates two datasets: Unicorn-1.2M and Unicorn-471K-Instruction. In Stage 1: Diverse Caption Data Synthesis, we construct 1.2M semantically diverse high-quality captions by expanding sparse caption seeds using large language models (LLMs). In Stage 2: Instruction-Tuning Data Generation, we further process 471K captions into multi-turn instruction-tuning tasks to support complex reasoning. Finally, in Stage 3: Modality Representation Transfer, these textual captions representations are transformed into visual representations, resulting in diverse synthetic image representations. This three-stage process enables us to construct Unicorn-1.2M for pretraining and Unicorn-471K-Instruction for instruction-tuning, without relying on real images. By eliminating the dependency on real images while maintaining data quality and diversity, our framework offers a cost-effective and scalable solution for VLMs training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。