用跨任务共识选出视觉语言模型关键数据，只用20%数据仍保持近100%性能。
ICONS: Influence Consensus for Vision-Language Data Selection
- 基于梯度影响度，通过多数投票整合多任务数据价值评分。
- 在三个数据集上仅用20%数据即保留98.6%~99.8%的模型性能。
- 适合需要高效训练多任务视觉语言模型的研究者使用。
通过指令微调训练视觉语言模型依赖于涵盖多种任务和领域的大型数据混合，但这些混合数据常包含冗余信息，增加计算成本却难以提升性能。现有方法通常依赖任务无关的启发式规则评估数据重要性，限制了其在不同任务中的有效性。我们提出ICONS，一种基于梯度的影响共识（Influence CONsensus）方法，用于视觉语言数据选择。该方法利用一阶训练动态估计每个样本对验证性能的影响，并通过多数投票在多个任务间聚合这些估计值。这种跨任务共识能识别出始终有价值的数据点，同时降低评分校准和异常值敏感性，实现对多样化多任务混合数据的鲁棒且可扩展的数据选择。在从LLAVA-665K中选取的20%数据子集上，模型保留了98.6%的完整数据集性能；类似地，在CAMBRIAN-7M和VISION-FLAN-186K上分别达到98.8%和99.8%。实验表明，所选数据具有泛化到未见任务和模型架构的能力，并发布了三个小型数据集：LLAVA-ICONS-133K、CAMBRIAN-ICONS-1.4M和VISION-FLAN-ICONS-37K，以支持高效的视觉语言模型开发。
原文摘要 · Abstract (English)
Training vision-language models via instruction tuning relies on large data mixtures spanning diverse tasks and domains, yet these mixtures frequently include redundant information that increases computational costs without proportional gains. Existing methods typically rely on task-agnostic heuristics to estimate data importance, limiting their effectiveness across tasks. We introduce ICONS, a gradient-based Influence CONsensus approach for vision-language data Selection. Our method leverages first-order training dynamics to estimate each example's influence on validation performance, then aggregates these estimates across tasks via majority voting. This cross-task consensus identifies consistently valuable data points while mitigating score calibration and outlier sensitivity, enabling robust and scalable data selection for diverse multitask mixtures. Models trained on our selected 20% data subset from LLAVA-665K (respectively: from CAMBRIAN-7M, from VISION-FLAN-186K) retain 98.6% (respectively: 98.8%, 99.8%) of full-dataset performance. We demonstrate that our selected data generalizes to unseen tasks and model architectures, and release three compact subsets LLAVA-ICONS-133K, CAMBRIAN-ICONS-1.4M, and VISION-FLAN-ICONS-37K for efficient vision-language model development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。