通过衡量视觉必要性，精选高效多模态指令数据。
VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuning
- 提出视觉必要性评分（VisNec），评估视觉输入的边际贡献。
- 仅用15%精选数据，性能达全量数据的100.2%；小数据集上还超全量训练15.8%。
- 适合追求高效、鲁棒多模态训练的研究者与工程师。
多模态指令微调的效果不仅取决于数据规模，更关键在于样本是否真正需要视觉推理。然而现有指令数据集包含大量仅靠文本即可解答的视觉冗余样本，以及多模态对齐错误的监督信号，会损害学习效果。为此，我们提出VisNec（视觉必要性评分），一种基于原则的数据筛选框架，通过对比有无视觉上下文时的预测损失，判断训练实例是否为视觉关键、冗余或错位。为保持任务多样性，将VisNec与语义聚类结合，在每类中选取高必要性样本。在10个下游基准测试中，仅使用15%的LLaVA-665K数据子集，即达到全数据性能的100.2%；在较小的Vision-Flan-186K数据集上，不仅进一步压缩数据量，还超越全量训练15.8%。结果表明，测量并利用视觉必要性，是实现高效且鲁棒多模态指令微调的有效方案。代码与精选子集将在接受后发布。
原文摘要 · Abstract (English)
The effectiveness of multimodal instruction tuning depends not only on dataset scale, but critically on whether training samples genuinely require visual reasoning. However, existing instruction datasets often contain a substantial portion of visually redundant samples (solvable from text alone), as well as multimodally misaligned supervision that can degrade learning. To address this, we propose VisNec (Visual Necessity Score), a principled data selection framework that measures the marginal contribution of visual input during instruction tuning. By comparing predictive loss with and without visual context, VisNec identifies whether a training instance is vision-critical, redundant, or misaligned. To preserve task diversity, we combine VisNec with semantic clustering and select high-necessity samples within each cluster. Across 10 downstream benchmarks, training on only 15% of the LLaVA-665K dataset selected by VisNec achieves 100.2% of full-data performance. On the smaller Vision-Flan-186K dataset, our selection not only further reduces data size but also surpasses full-data training by 15.8%. These results demonstrate that measuring and leveraging visual necessity provides an effective solution for both efficient and robust multimodal instruction tuning. Codes and selected subsets will be released upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。