arXiv:2608.03218cs.CVcs.AI2026-08

用自监督表征引导生成,让压缩数据集更符合预训练模型需求。

Self-Supervised Representation-Guided Generative Dataset Distillation

论文配图:Self-Supervised Representation-Guided Generative Dataset Distillation
图 1 · 摘自论文原文
  • 基于自监督表征构建类原型,通过三重目标指导生成过程。
  • 在多个数据集和样本量下均优于现有生成式方法。
  • 适合使用预训练编码器的视觉任务,提升迁移性能。

数据集蒸馏将大规模训练集压缩为紧凑的合成数据集,同时保持其下游效用。现有方法多针对随机初始化网络,而现代视觉系统常采用冻结的预训练编码器搭配轻量模块。因此,蒸馏样本应保留预训练表示空间中的判别性几何结构,但现有生成目标未显式考虑此点。本文提出自监督表征引导的生成式数据集蒸馏(SRG),将自监督学习(SSL)的几何结构转化为扩散模型的引导信号。具体地,SRG从真实图像的SSL表示中构建类内原型,并通过三重目标实现原型对齐、类间区分与类内分配。在扩散采样过程中,采用分阶段引导策略:早期去噪锚定于与目标原型最近的真实图像潜在表示,后期则由SSL空间目标引导。该设计在保持生成先验视觉真实性的同时,逐步将样本推向代表性强且类间可区分的SSL空间区域。SRG在多个数据集和每类样本数(IPC)设置下均优于对比的生成基线。跨编码器评估表明其在不同预训练表示空间间具备良好迁移能力。结果证明,以表征为导向的生成策略对基于预训练自监督模型的数据集蒸馏具有显著有效性。

原文摘要 · Abstract (English)

Dataset distillation compresses a large training set into a compact synthetic set while retaining its downstream utility. Most existing methods target randomly initialized networks, whereas modern vision systems often adapt frozen pretrained encoders with lightweight modules. Distilled samples should therefore preserve the discriminative geometry of the pretrained representation space, which existing generative objectives do not explicitly consider. We propose self-supervised representation-guided generative dataset distillation (SRG), a framework that translates the SSL geometry into diffusion guidance. Specifically, SRG constructs class-wise prototypes from real-image SSL representations and performs guidance through three SSL-space objectives for prototype alignment, inter-class discrimination, and intra-class assignment. During diffusion sampling, it adopts a stage-wise guidance strategy: early denoising is anchored to the latent of the real image whose SSL representation is nearest to the assigned prototype, whereas later denoising is guided by the SSL-space objectives. This division preserves the visual realism provided by the generative prior while progressively steering samples toward representative and class-discriminative regions of the SSL representation space. SRG consistently outperforms the evaluated generative baselines across multiple datasets and IPC settings. A cross-encoder evaluation further indicates transfer across pretrained representation spaces. These results demonstrate the effectiveness of representation-guided generation for dataset distillation with pretrained SSL models.

数据集蒸馏自监督学习生成模型预训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。