不训练模型,只靠问问题前后判断变化,选出真正需要图文联合理解的高质量数据。
Does the Question Really Matter? Training-Free Data Selection for Vision-Language SFT
- 用冻结的视觉语言模型评估提问前后的答案可信度差异
- 仅用10%数据就比全量数据高3.5%,在复杂数据集上也稳定有效
- 适合想高效提升图文模型性能的研究者和工程师
视觉指令微调对提升视觉语言大模型(VLLM)至关重要。然而,许多样本可通过语言模式或常识捷径解决,无需真正的跨模态推理,限制了多模态学习效果。现有数据选择方法依赖昂贵的代理模型训练,且关注难度或多样性,难以捕捉样本对图文联合推理的真实贡献。本文提出CVS,一种无需训练的数据选择方法:核心思想是高质量多模态样本中,加入问题会显著改变模型对答案有效性的判断。CVS利用冻结的VLLM作为评估器,测量有无问题时答案有效性的差异,从而识别需图文联合推理的样本,并过滤语义冲突噪声。在Vision-Flan和The Cauldron数据集上的实验表明,CVS表现优异;在Vision-Flan上,仅使用10%和15%数据即分别超越全量数据训练3.5%和4.8%,且在高度异构的Cauldron数据集上仍具鲁棒性。此外,相比COINCIDE和XMAS,CVS分别降低17.3%和44.4%的计算成本。
原文摘要 · Abstract (English)
Visual instruction tuning is crucial for improving vision-language large models (VLLMs). However, many samples can be solved via linguistic patterns or common-sense shortcuts, without genuine cross-modal reasoning, limiting the effectiveness of multimodal learning. Prior data selection methods often rely on costly proxy model training and focus on difficulty or diversity, failing to capture a sample's true contribution to vision-language joint reasoning. In this paper, we propose CVS, a training-free data selection method based on the insight that, for high-quality multimodal samples, introducing the question should substantially alter the model's assessment of answer validity given an image. CVS leverages a frozen VLLM as an evaluator and measures the discrepancy in answer validity with and without conditioning on the question, enabling the identification of samples that require vision-language joint reasoning while filtering semantic-conflict noise. Experiments on Vision-Flan and The Cauldron show that CVS achieves solid performance across datasets. On Vision-Flan, CVS outperforms full-data training by 3.5% and 4.8% using only 10% and 15% of the data, respectively, and remains robust on the highly heterogeneous Cauldron dataset. Moreover, CVS reduces computational cost by 17.3% and 44.4% compared to COINCIDE and XMAS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。