仅用一张图就能合成高质量多模态训练数据。
Oasis: One Image is All You Need for Multimodal Instruction Data Synthesis
- 仅用图像提示大模型生成多模态数据，突破传统依赖图文对的限制。
- 合成超50万条数据，使LLaVA-NeXT模型性能显著提升。
- 适合想低成本构建领域专用多模态模型的研究者使用。
多模态大模型的成功主要依赖大规模训练数据，但许多模型的数据因隐私问题无法公开，且收集多模态数据成本高、耗时长。能否在不牺牲多样性和质量的前提下自动合成多模态训练数据？本文提出Oasis方法，仅通过图像即可合成高质量多模态数据。该方法通过仅向多模态大模型输入图像进行提示，极大扩展了数据多样性。同时引入精细的质量控制机制，保障生成数据的可靠性。我们共收集超过50万条数据，并在LLaVA-NeXT上开展增量实验。大量实验证明，该方法可显著提升多模态大模型性能。基于图像的合成方式还便于聚焦特定领域的模型能力。代码与数据集已开源：https://github.com/Letian2003/MM_INF。
原文摘要 · Abstract (English)
The success of multi-modal large language models (MLLMs) has been largely attributed to the large-scale training data. However, the training data of many MLLMs is unavailable due to privacy concerns. The expensive and labor-intensive process of collecting multi-modal data further exacerbates the problem. Is it possible to synthesize multi-modal training data automatically without compromising diversity and quality? In this paper, we propose a new method, Oasis, to synthesize high-quality multi-modal data with only images. Oasis breaks through traditional methods by prompting only images to the MLLMs, thus extending the data diversity by a large margin. Our method features a delicate quality control method which ensures the data quality. We collected over 500k data and conducted incremental experiments on LLaVA-NeXT. Extensive experiments demonstrate that our method can significantly improve the performance of MLLMs. The image-based synthesis also allows us to focus on the specific-domain ability of MLLMs. Code and dataset are publicly available at https://github.com/Letian2003/MM_INF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。