让多模态模型自动生成训练数据并持续优化，无需人工干预。
MLLM-DataEngine: Closing the Loop of Multimodal Instruction Tuning Data Generation
- 基于评估结果自动分析模型弱点，动态生成针对性数据集。
- 通过上下文示例引导大模型生成高质量增量数据，提升能力更精准。
- 适合希望自动化构建多模态指令数据的研究者与开发者。
本文提出 MLLM-DataEngine，一个闭环系统，实现数据生成、模型训练与评估的协同迭代。每轮循环中，系统根据评估结果分析模型短板，生成针对性增量数据集，推动模型能力持续增强。相比以往分离式数据收集方法，该系统更具针对性，能更高效提升多模态大模型（MLLM）性能。首先，提出自适应劣化样本采样模块，可依据基准测试结果灵活调整数据生成策略；其次，为确保特定能力类型的数据质量，向 GPT-4 提供最具代表性的上下文示例与丰富信息，使其充分理解模型缺陷，从而保障生成数据的高质性。大量实验表明，该系统可无须人工参与，自动、定向地提升多模态大模型能力。代码、数据与模型已开源：https://github.com/opendatalab/MLLM-DataEngine。
原文摘要 · Abstract (English)
In this paper, we propose MLLM-DataEngine, a novel closed-loop system that bridges data generation, model training, and evaluation. Within each loop iteration, the MLLM-DataEngine first analyzes the weakness of the model based on the evaluation results, then generates a proper incremental dataset for the next training iteration, and enhances the model capability iteratively. Compared with previous instruction fine-tuning dataset collection methods which are separate from the benchmarking, MLLM-DataEngine shows better targeting and can improve MLLMs's capabilities more effectively. Firstly, we propose an Adaptive Bad-case Sampling module, which can effectively analyze model weakness based on the benchmarking results and adjust the generation of incremental datasets flexibly. Secondly, in order to ensure high-quality data for specific capability types, the most representative in-context examples and abundant information are provided to GPT-4, which helps GPT-4 fully comprehend the model's weakness and further guarantees high-quality generated data. Through extensive experiments, we find MLLM-DataEngine could boost the MLLMs capability in a targeted and automatic manner without human participants. We hope MLLM-DataEngine could be a general solution for the following MLLMs data curation. Code, data, and model are available at https://github.com/opendatalab/MLLM-DataEngine.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。