用概率潜空间特征提升数据蒸馏,生成更优合成图像
Dataset Distillation with Probabilistic Latent Features
- 建模潜空间特征的联合分布,通过低秩正态分布生成多样样本
- 在ImageNet子集、CIFAR-10等上实现跨架构最优性能
- 适合需要高效训练且关注合成数据质量的研究者
随着深度学习模型复杂度和训练数据量的增长,降低存储与计算成本变得愈发重要。数据蒸馏通过合成一组紧凑的合成数据来替代原始数据集,以支持下游分类任务。现有方法通常将数据从像素空间映射到生成模型的潜空间,本文提出一种新型随机方法,建模潜特征的联合分布。该方法能更好捕捉空间结构,并生成更具多样性的合成样本,从而提升模型训练效果。具体而言,我们引入一个由轻量网络参数化的低秩多元正态分布,保持低计算开销,且兼容各类数据蒸馏中的匹配网络。蒸馏完成后,将学习到的潜特征输入预训练生成器,生成合成图像用于分类模型训练,并在真实测试集上评估性能。我们在多个基准上验证方法,包括ImageNet子集、CIFAR-10和MedMNIST病理图像数据集。结果表明,该方法在多种主干网络架构下均实现跨架构最优表现,证明其通用性与有效性。
原文摘要 · Abstract (English)
As deep learning models grow in complexity and the volume of training data increases, reducing storage and computational costs becomes increasingly important. Dataset distillation addresses this challenge by synthesizing a compact set of synthetic data that can effectively replace the original dataset in downstream classification tasks. While existing methods typically rely on mapping data from pixel space to the latent space of a generative model, we propose a novel stochastic approach that models the joint distribution of latent features. This allows our method to better capture spatial structures and produce diverse synthetic samples, which benefits model training. Specifically, we introduce a low-rank multivariate normal distribution parameterized by a lightweight network. This design maintains low computational complexity and is compatible with various matching networks used in dataset distillation. After distillation, synthetic images are generated by feeding the learned latent features into a pretrained generator. These synthetic images are then used to train classification models, and performance is evaluated on real test set. We validate our method on several benchmarks, including ImageNet subsets, CIFAR-10, and the MedMNIST histopathological dataset. Our approach achieves state-of-the-art cross architecture performance across a range of backbone architectures, demonstrating its generality and effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。