新方法CIM直接压缩数据集,信息损失更少,训练更快。
Condensing Large-Scale Datasets Directly with Minimal Information Loss

- 不走传统两步压缩路径,直接对齐原始与合成数据分布
- 在单张4090显卡上80分钟完成ImageNet-1K蒸馏,精度达48.7%
- 适合追求高效、高保真数据压缩的研究者和工业应用
当前大规模数据蒸馏依赖解耦的提取-重建-重标注流程,虽可扩展至大规模数据,但计算开销巨大且跨架构泛化差。本文揭示其根本瓶颈在于隐含的双重压缩过程:从数据到模型再回退至图像,导致严重信息损失。实证与理论证明,这种损失引发分布偏移,使预训练模型无法可靠生成标签,从而损害重标注策略效果。为此,我们提出CIM框架,摒弃双压缩范式,以度量驱动方式显式量化并最小化原数据与合成数据间的差距。通过直接对齐数据分布,实现高保真信息压缩,并自然满足有效重标注的前提条件。大量实验表明,CIM达到新SOTA:在单块RTX-4090 GPU上仅用80分钟即完成ImageNet-1K蒸馏(IPC=10),ResNet-18上取得48.7% Top-1准确率,显著优于NRR-DD和DELT,分别提升2.6%和2.9%。
原文摘要 · Abstract (English)
Recent advancements in scaling dataset distillation rely heavily on decoupled information extraction pipelines, comprising SQUEEZE, RECOVER, and RELABEL stages. Despite their scalability to large-scale datasets, these methods suffer from prohibitive computational overhead and poor cross-architecture generalization. In this paper, we reveal the root cause of these bottlenecks: the implicit dual-compression process, from data to model and back to images, inherently induces severe information loss. Crucially, we empirically and theoretically demonstrate that this loss creates a distribution shift that fundamentally compromises the widely adopted RELABEL strategy, transforming the pre-trained model into an unreliable labeler that yields sub-optimal labels. To overcome these critical flaws, we propose CIM, a novel, metric-driven framework that abandons the flawed dual-compression paradigm. Instead, CIM explicitly quantifies and minimizes the information gap between the original and synthetic datasets. By directly aligning the data distributions, our approach ensures high-fidelity information condensation and inherently satisfies the prerequisites for effective relabeling. Extensive experiments demonstrate that CIM establishes a new state-of-the-art. Notably, it distills ImageNet-1K at an IPC=10 in merely 80 minutes on a single RTX-4090 GPU, achieving an unprecedented 48.7% Top-1 accuracy on ResNet-18 and significantly outperforming previous SOTA approaches, such as NRR-DD and DELT, by 2.6% and 2.9%, respectively. Our code is available at https://github.com/LINs-lab/CIM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。