用统计流匹配高效压缩预训练模型的数据集,速度快10倍,内存少4倍。
Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matching
- 通过对齐真实数据的类中心统计流优化合成图像,避免反复加载真实图像。
- 仅需一次数据增强,即可在ImageNet上达到当前最佳性能,内存降低10倍。
- 复用原数据集分类器,只需轻量投影头,适合资源受限场景部署。
数据集蒸馏旨在生成一个高度紧凑的数据集,使其在下游任务上的表现可媲美原始数据集。针对以预训练自监督模型为骨干的分类任务,以往的线性梯度匹配方法通过使合成图像模仿真实图像在分类器上的梯度更新来优化,但该方法需每步加载数千张真实图像并进行多轮可微增强,带来巨大计算与内存开销。本文提出统计流匹配(statistical flow matching),一种稳定高效的监督学习框架,通过将合成图像的统计流对齐至真实数据中目标类中心到非目标类中心的恒定统计流来优化。本方法仅需一次性加载原始统计数据,对合成数据执行单次增强,即可在性能上达到或优于现有最优方法,实现10倍的GPU内存节省和4倍的运行时间缩短。此外,我们提出分类器继承策略,复用原始数据集训练的分类器进行推理,仅需极轻量级线性投影器和极少存储,便能获得显著性能提升。
原文摘要 · Abstract (English)
Dataset distillation seeks to synthesize a highly compact dataset that achieves performance comparable to the original dataset on downstream tasks. For the classification task that use pre-trained self-supervised models as backbones, previous linear gradient matching optimizes synthetic images by encouraging them to mimic the gradient updates induced by real images on the linear classifier. However, this batch-level formulation requires loading thousands of real images and applying multiple rounds of differentiable augmentations to synthetic images at each distillation step, leading to substantial computational and memory overhead. In this paper, we introduce statistical flow matching , a stable and efficient supervised learning framework that optimizes synthetic images by aligning constant statistical flows from target class centers to non-target class centers in the original data. Our approach loads raw statistics only once and performs a single augmentation pass on the synthetic data, achieving performance comparable to or better than the state-of-the-art methods with 10x lower GPU memory usage and 4x shorter runtime. Furthermore, we propose a classifier inheritance strategy that reuses the classifier trained on the original dataset for inference, requiring only an extremely lightweight linear projector and marginal storage while achieving substantial performance gains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。