用最优传输优化生成数据集蒸馏,提升模型泛化能力。
Optimizing Distributional Geometry Alignment with Optimal Transport for Generative Dataset Distillation
- 基于最优传输框架,同时对齐全局与实例级分布几何。
- 在ImageNet-1K上,每种架构均实现至少4%的准确率提升(IPC=10)。
- 适合追求高效高精度数据蒸馏的研究者和工业应用。
数据集蒸馏旨在生成紧凑的合成数据集,使在该数据集上训练的模型性能可媲美在完整数据集上训练的模型。现有大规模数据集方法主要关注匹配全局分布统计量(如均值和方差),却忽略了关键的实例级特征和类内变化,导致泛化性能不佳。本文将数据集蒸馏重新建模为最优传输(OT)距离最小化问题,实现了在整个流程中从全局到实例级别的精细对齐。OT提供了一个几何忠实的分布匹配框架,能有效保留局部模式、类内结构及高维复杂分布中的细微变化。所提方法包含三个组件:(1) 基于OT的扩散采样,对齐真实与蒸馏图像的潜在分布;(2) 标签-图像对齐的软重标注,根据蒸馏图像分布的复杂度调整标签分布;(3) 基于OT的输出对齐,使学生模型的输出与软标签分布一致。在多种架构和大规模数据集上的实验表明,本方法在效率和性能上均优于现有最先进方法,在ImageNet-1K上,各架构在IPC=10设置下均获得至少4%的准确率提升。
原文摘要 · Abstract (English)
Dataset distillation seeks to synthesize a compact distilled dataset, enabling models trained on it to achieve performance comparable to models trained on the full dataset. Recent methods for large-scale datasets focus on matching global distributional statistics (e.g., mean and variance), but overlook critical instance-level characteristics and intraclass variations, leading to suboptimal generalization. We address this limitation by reformulating dataset distillation as an Optimal Transport (OT) distance minimization problem, enabling fine-grained alignment at both global and instance levels throughout the pipeline. OT offers a geometrically faithful framework for distribution matching. It effectively preserves local modes, intra-class patterns, and fine-grained variations that characterize the geometry of complex, high-dimensional distributions. Our method comprises three components tailored for preserving distributional geometry: (1) OT-guided diffusion sampling, which aligns latent distributions of real and distilled images; (2) label-image-aligned soft relabeling, which adapts label distributions based on the complexity of distilled image distributions; and (3) OT-based logit matching, which aligns the output of student models with soft-label distributions. Extensive experiments across diverse architectures and large-scale datasets demonstrate that our method consistently outperforms state-of-the-art approaches in an efficient manner, achieving at least 4% accuracy improvement under IPC=10 settings for each architecture on ImageNet-1K.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。