arXiv:2502.19917cs.CL2025-02被引 2

用智能协作筛选高质量图文数据,仅用2.5%数据超越主流模型。

Picking the Cream of the Crop: Visual-Centric Data Selection with Collaborative Agents

  • 通过视觉代理协作评估图像信息量,筛选高信息密度图片。
  • 基于图像质量评估指令相关性,剔除低质或无关指令数据。
  • 在7个基准上仅用2.5%原始数据即达顶尖性能,适合数据高效训练场景。

为提升多模态大语言模型处理图像与复杂指令的能力,研究者通常构建大规模视觉指令调优数据集,但这些数据常存在图文不匹配、图像质量差等问题,导致训练效率低且性能提升有限。为此,我们提出视觉中心的数据选择方法ViSA,通过代理协作实现图像质量评估与图文相关性判断。具体包括:1)利用视觉代理协作量化图像信息量,选取富含视觉信息的图片;2)基于图像质量评估指令质量,筛选与高质量图像相关的优质指令数据。最终对80K条开源指令数据进行重构。大量实验表明,ViSA在7个基准上表现优于或相当于当前最优模型,仅使用原始数据的2.5%,充分验证了其高效性。消融实验进一步证实各组件有效性。代码已公开于https://github.com/HITsz-TMG/ViSA。

原文摘要 · Abstract (English)

To improve Multimodal Large Language Models' (MLLMs) ability to process images and complex instructions, researchers predominantly curate large-scale visual instruction tuning datasets, which are either sourced from existing vision tasks or synthetically generated using LLMs and image descriptions. However, they often suffer from critical flaws, including misaligned instruction-image pairs and low-quality images. Such issues hinder training efficiency and limit performance improvements, as models waste resources on noisy or irrelevant data with minimal benefit to overall capability. To address this issue, we propose a \textbf{Vi}sual-Centric \textbf{S}election approach via \textbf{A}gents Collaboration (ViSA), which centers on image quality assessment and image-instruction relevance evaluation. Specifically, our approach consists of 1) an image information quantification method via visual agents collaboration to select images with rich visual information, and 2) a visual-centric instruction quality assessment method to select high-quality instruction data related to high-quality images. Finally, we reorganize 80K instruction data from large open-source datasets. Extensive experiments demonstrate that ViSA outperforms or is comparable to current state-of-the-art models on seven benchmarks, using only 2.5\% of the original data, highlighting the efficiency of our data selection approach. Moreover, we conduct ablation studies to validate the effectiveness of each component of our method. The code is available at https://github.com/HITsz-TMG/ViSA.

数据筛选多模态智能代理高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。