arXiv:2605.26761cs.CV2026-05被引 2

训练一次选择器,可通用所有模型和数据集,大幅减少冗余数据。

Once-For-All: A Train-Once and Select-Anytime Framework for Multimodal Instruction Tuning

论文配图:Once-For-All: A Train-Once and Select-Anytime Framework for Multimodal Instruction Tuning
图 1 · 摘自论文原文
  • 在冻结的CLIP空间聚类指令,用伪标签训练轻量选择器。
  • 仅选15%数据即达98.3%全量性能,跨数据集提升10.6%。
  • 一次训练,跨模型、跨数据集直接复用,无需重算。

多模态指令微调是视觉语言模型(VLMs)适配的标准方法,但指令数据高度冗余,数据选择对训练效率至关重要。现有方法依赖特定模型或数据集生成选择信号,一旦目标模型或候选池变化,必须重新计算,成本高昂。为此,我们提出OFA框架:训练一次可复用的选择器,适用于任意数据集或模型而无需重新计算。OFA在冻结的CLIP空间中对多模态指令进行聚类,基于聚类结构生成伪标签,仅用少数几个周期训练轻量选择器;选择器置信度最低的样本被判定为最信息量的样本。训练完成后,该冻结选择器可直接跨数据集与模型规模迁移。选择器在LLaVA-665K上训练,应用于自身及未见的Vision-Flan-186K,仅选取15%数据即在10个下游基准上达到98.3%的全量性能;在较小的Vision-Flan-186K上,迁移选择器表现优于全量训练10.6%,证明所学信号可泛化至训练阶段未见的数据集。相同选子集同时适用于Qwen2.5-VL-3B与LLaVA-v1.5-7B,无需针对每个模型重新计算,实现选择与目标模型解耦。结果表明,单一可迁移选择器为高效多模态指令微调提供了有效且可复用的解决方案。

原文摘要 · Abstract (English)

Multimodal instruction tuning is the de facto recipe for adapting vision language models (VLMs), yet instruction data are highly redundant, making data selection critical for training efficiency. Existing methods derive selection signals from a specific model or dataset, so whenever the target model or candidate pool changes, the criteria must be recomputed from scratch at substantial cost. To address this, we propose OFA, a data selection framework that trains a reusable selector once and applies it to any dataset or model without recomputation. OFA clusters multimodal instructions in a frozen CLIP space, derives pseudo labels from the cluster structure, and trains a lightweight selector for only a few epochs; samples on which this selector is least confident are selected as the most informative. Once trained, the frozen selector transfers directly across datasets and model scales. The selector is trained once on LLaVA-665K and applied both to LLaVA-665K itself and, without any retraining, to the unseen Vision-Flan-186K. Selecting only 15% of the data, OFA achieves 98.3% of full data performance across 10 downstream benchmarks; on the smaller Vision-Flan-186K, the transferred selector surpasses full data training by 10.6%, confirming that the learned signal generalizes to datasets never seen during selector training. The same selected subsets benefit VLMs at both Qwen2.5-VL-3B and LLaVA-v1.5-7B without per model recomputation, decoupling selection from the target model. These results demonstrate that a single, transferable selector provides an effective and reusable solution for efficient multimodal instruction tuning.

多模态指令微调数据选择迁移学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。