arXiv:2502.12119cs.CVcs.AI2025-02中稿 · EMNLP被引 72

提出无需训练的高效多模态数据筛选方法,显著提升模型训练速度与性能。

PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection

  • 通过隐式重中心化建模视觉语义,消除背景噪声干扰
  • 数据筛选与模型调优总时长降至传统方法的30%
  • 在8个视觉理解任务上超越全量数据微调模型

视觉指令微调旨在使预训练多模态大模型(MLLMs)适应真实场景中的人类指令。然而,数据集快速增长带来显著冗余,导致计算成本激增。现有数据选择方法多依赖高耗能的代理推理或训练型指标,其自身带来的计算开销反而加剧了效率瓶颈,制约了MLLMs的可扩展性与有效性。我们首次识别出一个关键但被忽视的因素:视觉特征分布的各向异性。该特性引发全局语义漂移,是当前数据选择方法效率受限的核心原因。基于此,我们提出首个无需训练的高效视觉指令选择框架PRISM。PRISM通过隐式重中心化建模内在视觉语义,精准剔除全局背景特征的污染影响。实验表明,PRISM将数据筛选与模型调优的端到端时间压缩至传统流程的30%;更显著的是,在八个多模态和三个语言理解基准上,其性能超越全量数据微调模型,相对基线提升达101.7%。代码已开源。

原文摘要 · Abstract (English)

Visual instruction tuning adapts pre-trained Multimodal Large Language Models (MLLMs) to follow human instructions for real-world applications. However, the rapid growth of these datasets introduces significant redundancy, leading to increased computational costs. Existing methods for selecting instruction data aim to prune this redundancy, but predominantly rely on computationally demanding techniques such as proxy-based inference or training-based metrics. Consequently, the substantial computational costs incurred by these selection processes often exacerbate the very efficiency bottlenecks they are intended to resolve, posing a significant challenge to the scalable and effective tuning of MLLMs. To address this challenge, we first identify a critical, yet previously overlooked, factor: the anisotropy inherent in visual feature distributions. We find that this anisotropy induces a \textit{Global Semantic Drift}, and overlooking this phenomenon is a key factor limiting the efficiency of current data selection methods. Motivated by this insight, we devise \textbf{PRISM}, the first training-free framework for efficient visual instruction selection. PRISM surgically removes the corrupting influence of global background features by modeling the intrinsic visual semantics via implicit re-centering. Empirically, PRISM reduces the end-to-end time for data selection and model tuning to just 30\% of conventional pipelines. More remarkably, it achieves this efficiency while simultaneously enhancing performance, surpassing models fine-tuned on the full dataset across eight multimodal and three language understanding benchmarks, culminating in a 101.7\% relative improvement over the baseline. The code is available for access via \href{https://github.com/bibisbar/PRISM}{this repository}.

多模态数据筛选训练自由效率优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。