用小型视觉语言模型自动筛选图像文本数据质量,效果媲美大规模清洗。
Trust the Model: Compact VLMs as In-Context Judges for Image-Text Data Quality
- 用微调的小型VLM评估图像与文字的匹配度和质量。
- 过滤后数据在对齐性和语义流畅性上优于大量噪声数据。
- 适合需要高质量数据集的研究者快速构建训练集。
视觉语言模型(VLM)通过整合视觉信息拓展了大语言模型的能力,支持更丰富的多模态推理,应用范围显著扩大。然而,引入视觉输入也带来了数据质量维护的新挑战。实证表明,精心筛选且具有代表性的训练样本往往比单纯增加数据量更有效。受此启发,我们提出一种轻量级数据过滤框架,采用在高质量图像-标题标注数据集上微调的紧凑型VLM,基于标题与图像的质量及对齐程度评估并筛选潜在训练样本。不同于以往在大型VLM上叠加额外过滤模块的方法,本方法仅利用专为评估设计的小型VLM的内在判别能力,无需额外模块,降低训练开销。该轻量模型能高效剔除不准确、含噪的网络数据,提升图像-文本对齐度与标题语言流畅性。实验表明,经该紧凑VLM高精度过滤后的数据集性能达到甚至超越通过大规模网络爬取获得的大规模、嘈杂数据集。因此,该方法提供了一种轻量而稳健的高质量视觉语言训练语料构建方案。
原文摘要 · Abstract (English)
Vision-language models (VLMs) extend the conventional large language models by integrating visual data, enabling richer multimodal reasoning and significantly broadens the practical applications of AI. However, including visual inputs also brings new challenges in maintaining data quality. Empirical evidence consistently shows that carefully curated and representative training examples often yield superior results compared to simply increasing the quantity of data. Inspired by this observation, we introduce a streamlined data filtration framework that employs a compact VLM, fine-tuned on a high-quality image-caption annotated dataset. This model effectively evaluates and filters potential training samples based on caption and image quality and alignment. Unlike previous approaches, which typically add auxiliary filtration modules on top of existing full-scale VLMs, our method exclusively utilizes the inherent evaluative capability of a purpose-built small VLM. This strategy eliminates the need for extra modules and reduces training overhead. Our lightweight model efficiently filters out inaccurate, noisy web data, improving image-text alignment and caption linguistic fluency. Experimental results show that datasets underwent high-precision filtration using our compact VLM perform on par with, or even surpass, larger and noisier datasets gathered through high-volume web crawling. Thus, our method provides a lightweight yet robust solution for building high-quality vision-language training corpora. \\ \textbf{Availability and implementation:} Our compact VLM filtration model, training data, utility scripts, and Supplementary data (Appendices) are freely available at https://github.com/daulettoibazar/Compact_VLM_Filter.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。