arXiv:2605.26004cs.CVcs.CL2026-05

MAGIC通过三重信号筛选,用更少数据提升视觉语言模型训练效果。

MAGIC: Multimodal Alignment & Grounding-aware Instruction Coreset for Vision-Language Models

论文配图:MAGIC: Multimodal Alignment & Grounding-aware Instruction Coreset for Vision-Language Models
图 1 · 摘自论文原文
  • 基于多模态增益、连接相关性和技能神经签名三类信号筛选数据。
  • 在20%数据下性能达全量微调的100.3%~101.6%,提速73.7%。
  • 无需反向传播,可直接部署于现有视觉语言模型中。

大型视觉语言模型(LVLM)的指令微调依赖海量多模态语料,但这些数据存在冗余、视觉依赖弱、多模态推理行为覆盖不均等问题。传统均匀采样或基于分数的选择往往效果不佳。本文提出MAGIC,一种无需训练、仅前向传播的共集选择方法,旨在构建紧凑且行为忠实的多模态指令微调子集。MAGIC基于预训练视觉语言模型提取三个内在信号:多模态增益(衡量视觉输入带来的性能提升)、桥接相关性(捕捉答案词元对视觉词元的定位精确度)、技能神经签名(通过最高激活前馈神经元表征样本诱发的功能计算)。该方法采用三阶段流程:过滤低增益样本,按归一化质量目标排序候选样本,并基于离散神经签名进行分桶预算分配,以保持潜在多模态技能覆盖。该方法避免反向传播、辅助选择器训练及连续激活空间聚类,高效且易于集成。在LLaVA-665K与Vision-Flan数据集上,以及迁移至大目标模型LLaVA-1.5-7B和-13B时,MAGIC在20%预算下持续优于强基线:在LLaVA-665K上达到全量微调100.3%的相对性能,在Vision-Flan-186K上达101.6%,同时节省73.7%的运行时间。

原文摘要 · Abstract (English)

Instruction tuning of large vision-language models (LVLMs) increasingly depends on massive multimodal corpora, yet these datasets contain samples with substantial redundancy, low visual dependency, and highly imbalanced coverage of multimodal reasoning behaviors. As a result, uniform subsampling or naive score-based selection often yields suboptimal training subsets. We introduce MAGIC, a training-free, forward-only coreset selection method designed to construct compact yet behaviorally faithful subsets for multimodal instruction tuning. MAGIC is built on three intrinsic signals extracted from a pretrained VLM: Multimodal Gain, which measures the likelihood improvement obtained from visual input; Bridging Relevance, which captures the sharpness of answer-token grounding over visual tokens; and Skill-Neuron Signatures, which characterize the functional computation elicited by each sample via top-activated feed-forward neurons. MAGIC combines these signals in a three-stage pipeline: filtering low-gain examples, ranking candidates by a normalized quality objective, and performing bucket-wise budget allocation over discrete neuron signatures to preserve latent multimodal skill coverage. This formulation avoids backpropagation, auxiliary selector training, and expensive clustering in continuous activation spaces, while remaining efficient and easily deployable in existing VLMs. Across LLaVA-665K and Vision-Flan datasets, and transfer settings to large target models, LLaVA-1.5-7B and -13B, MAGIC consistently improves over strong baselines under matched 20% budgets: it achieves 100.3% relative performance to full finetuning on LLaVA-665K and 101.6% relative performance on Vision-Flan-186K, while yielding a 73.7% reduction in wall-clock run time.

视觉语言模型数据筛选指令微调共集选择

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。