用三原则筛选多模态数据，少用数据也能达到顶尖效果
Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativeness
- 基于信息量、独特性、代表性三原则自动筛选数据
- 仅用15%数据即达全量数据101.3%的性能
- 适合追求高效训练的多模态模型研究者
指令微调用于提升预训练多模态大模型（MLLMs）在真实任务中的表现。然而，视觉指令数据集的快速扩展带来了数据冗余，导致计算成本过高。本文提出协同框架DataTailor，基于信息量、独特性和代表性三个核心原则进行有效数据选择。我们认为有价值的样本应具备任务相关信息、非冗余且能代表数据分布（即非异常值）。我们进一步提出了针对每项原则的实用评分方法，可自动适配给定数据集，无需繁琐超参数调优。在多个基准测试上的全面实验表明，DataTailor仅使用15%的数据即可达到全量数据微调101.3%的性能，显著降低计算成本的同时保持优异结果。这体现了多模态大模型发展中“少即是多”的理念。代码与数据见：https://github.com/Yuqifan1117/DataTailor。
原文摘要 · Abstract (English)
Instruction tuning fine-tunes pre-trained Multi-modal Large Language Models (MLLMs) to handle real-world tasks. However, the rapid expansion of visual instruction datasets introduces data redundancy, leading to excessive computational costs. We propose a collaborative framework, DataTailor, which leverages three key principles--informativeness, uniqueness, and representativeness--for effective data selection. We argue that a valuable sample should be informative of the task, non-redundant, and represent the sample distribution (i.e., not an outlier). We further propose practical ways to score against each principle, which automatically adapts to a given dataset without tedious hyperparameter tuning. Comprehensive experiments on various benchmarks demonstrate that DataTailor achieves 101.3% of the performance of full-data fine-tuning with only 15% of the data, significantly reducing computational costs while maintaining superior results. This exemplifies the "Less is More" philosophy in MLLM development. The code and data is available in this \href{https://github.com/Yuqifan1117/DataTailor}{URL}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。