通过动态优化图文配对质量，提升多模态大模型训练效率。
Beyond Filtering: Adaptive Image-Text Quality Enhancement for MLLM Pretraining
- 动态评估并重写低质图文对，保留原始数据量
- 引入负样本学习策略，显著提升质量判别能力
- 适合大规模多模态预训练数据优化场景
多模态大语言模型（MLLM）在融合视觉与文本模态方面取得显著进展。训练MLLM的关键在于多模态预训练数据集中图文对的质量。然而，现有基于过滤的数据质量增强方法常因图文语义对齐不足而丢弃大量高质量图像数据，导致数据利用率和可扩展性低下。本文提出自适应图文质量增强器（AITQE），一种动态评估并提升图文对质量的模型。AITQE采用文本重写机制处理低质量图文对，并在训练中融入精心挑选的低质量样本，以负样本学习方式提升评估能力。与以往显著改变文本分布的方法不同，本方法仅做最小化文本调整，有效保持数据总量的同时提升质量。实验表明，AITQE在多个基准上优于现有方法，能高效利用原始数据，并随数据量增加实现良好扩展。代码与模型已开源：https://github.com/hanhuang22/AITQE。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have made significant strides by integrating visual and textual modalities. A critical factor in training MLLMs is the quality of image-text pairs within multimodal pretraining datasets. However, $\textit {de facto}$ filter-based data quality enhancement paradigms often discard a substantial portion of high-quality image data due to inadequate semantic alignment between images and texts, leading to inefficiencies in data utilization and scalability. In this paper, we propose the Adaptive Image-Text Quality Enhancer (AITQE), a model that dynamically assesses and enhances the quality of image-text pairs. AITQE employs a text rewriting mechanism for low-quality pairs and incorporates a negative sample learning strategy to improve evaluative capabilities by integrating deliberately selected low-quality samples during training. Unlike prior approaches that significantly alter text distributions, our method minimally adjusts text to preserve data volume while enhancing quality. Experimental results demonstrate that AITQE surpasses existing methods on various benchmark, effectively leveraging raw data and scaling efficiently with increasing data volumes. We hope our work will inspire future works. The code and model are available at: https://github.com/hanhuang22/AITQE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。