用信息最大化方法挑选高价值数据，减少冗余，提升模型训练效率。
Data Pruning by Information Maximization
- 基于样本重要性与相似性，构建信息最大化选择模型。
- 在图像分类、多模态预训练等任务中显著提升数据利用率。
- 适用于大规模数据集，适合需要高效数据筛选的研究者。
本文提出InfoMax，一种新型数据剪枝方法（即coreset选择），旨在最大化所选样本的信息量同时最小化冗余。通过重要性得分衡量单个样本的影响力或学习难度，利用成对样本相似性量化冗余，将coreset选择建模为离散二次规划（DQP）问题：目标是最大化总信息量，即个体贡献之和减去相似样本带来的冗余。为保证可扩展性，引入基于梯度的高效求解器，并结合相似性矩阵稀疏化与数据集分块策略，使InfoMax能无缝处理百万级样本。大量实验表明，InfoMax在图像分类、视觉-语言预训练及大语言模型指令微调等多种数据剪枝任务中表现优异。代码已开源：https://github.com/hrtan/InfoMax。
原文摘要 · Abstract (English)
In this paper, we present InfoMax, a novel data pruning method, also known as coreset selection, designed to maximize the information content of selected samples while minimizing redundancy. By doing so, InfoMax enhances the overall informativeness of the coreset. The information of individual samples is measured by importance scores, which capture their influence or difficulty in model learning. To quantify redundancy, we use pairwise sample similarities, based on the premise that similar samples contribute similarly to the learning process. We formalize the coreset selection problem as a discrete quadratic programming (DQP) task, with the objective of maximizing the total information content, represented as the sum of individual sample contributions minus the redundancies introduced by similar samples within the coreset. To ensure practical scalability, we introduce an efficient gradient-based solver, complemented by sparsification techniques applied to the similarity matrix and dataset partitioning strategies. This enables InfoMax to seamlessly scale to datasets with millions of samples. Extensive experiments demonstrate the superior performance of InfoMax in various data pruning tasks, including image classification, vision-language pre-training, and instruction tuning for large language models. Code is available at https://github.com/hrtan/InfoMax.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。