用必要性与多样性筛选高质量视觉指令数据，少用数据也能提升模型表现。
MLLM-Selector: Necessity and Diversity-driven High-Value Data Selection for Enhanced Visual Instruction Tuning
- 基于种子模型评估数据必要性，结合多样性策略筛选高价值样本。
- 仅用不到1%数据就超越LLaVA-1.5，低于50%数据时全面领先所有基准。
- 适合追求高效训练、资源受限的视觉语言模型优化场景。
视觉指令微调（VIT）已成为多模态大语言模型（MLLMs）有效理解用户指令的关键技术。然而，对高质量指令微调数据特征及自动化选择框架的理解仍不充分。为此，我们提出MLLM-Selector，一种通过权衡必要性与多样性的自动化数据筛选方法。首先从VIT数据池中随机采样子集，微调预训练模型以获得具备初步指令理解能力的种子模型；随后利用该种子模型为数据池中每条样本计算必要性得分，识别对性能提升至关重要的样本。研究发现，将必要性与多样性结合可显著提升数据选择效果，由此构建了MLLM-Selector方法——融合必要性评分与策略性采样的数据精炼机制。实证结果表明，在相同实验条件下，使用不足1%的数据量即超越LLaVA-1.5，而使用低于50%数据时，在所有验证基准上均实现持续领先。
原文摘要 · Abstract (English)
Visual instruction tuning (VIT) has emerged as a crucial technique for enabling multi-modal large language models (MLLMs) to follow user instructions adeptly. Yet, a significant gap persists in understanding the attributes of high-quality instruction tuning data and frameworks for its automated selection. To address this, we introduce MLLM-Selector, an automated approach that identifies valuable data for VIT by weighing necessity and diversity. Our process starts by randomly sampling a subset from the VIT data pool to fine-tune a pretrained model, thus creating a seed model with an initial ability to follow instructions. Then, leveraging the seed model, we calculate necessity scores for each sample in the VIT data pool to identify samples pivotal for enhancing model performance. Our findings underscore the importance of mixing necessity and diversity in data choice, leading to the creation of MLLM-Selector, our methodology that fuses necessity scoring with strategic sampling for superior data refinement. Empirical results indicate that within identical experimental conditions, MLLM-Selector surpasses LLaVA-1.5 in some benchmarks with less than 1% of the data and consistently exceeds performance across all validated benchmarks when using less than 50%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。