arXiv:2503.18267cs.CV2025-03CVPR被引 12

通过精修非关键区域,让合成数据同时保留细节与类别共性,提升压缩效果。

Enhancing Dataset Distillation via Non-Critical Region Refinement

  • 区分实例特有特征与类别通用特征,分别优化合成数据的细节与共性
  • 在小规模和大规模数据集上均达当前最佳性能,仅存每例两个距离即可
  • 无需软标签,适合对压缩效率和泛化能力要求高的场景

数据集蒸馏近年来成为将大数据集压缩为高效小规模表示的热门方法,旨在保留关键信息用于模型训练。数据特征可分为两类:实例特有特征(捕捉单个样本的细微差异)和类别通用特征(反映类间共享的宏观模式)。现有方法常在二者间失衡:有的只关注类别共性,忽略个体细节;有的则过度强调实例特征,忽视类级理解所需的共性。本文提出非关键区域精修数据集蒸馏(NRR-DD),在保持合成数据中实例特有细节和细粒度区域的同时,向非关键区域注入类别通用信息,使模型能利用全部像素信息,兼顾两种特征,从而提升整体性能。此外,我们引入基于距离的代表性知识迁移(DBR),通过合成数据预测与一热编码标签间的距离替代软标签训练,无需额外标注。实验表明,NRR-DD在小规模与大规模数据集上均达到领先性能;且每实例仅需存储两个距离,即可在多种设置下保持良好表现。代码已开源:https://github.com/tmtuan1307/NRR-DD。

原文摘要 · Abstract (English)

Dataset distillation has become a popular method for compressing large datasets into smaller, more efficient representations while preserving critical information for model training. Data features are broadly categorized into two types: instance-specific features, which capture unique, fine-grained details of individual examples, and class-general features, which represent shared, broad patterns across a class. However, previous approaches often struggle to balance these features-some focus solely on class-general patterns, neglecting finer instance details, while others prioritize instance-specific features, overlooking the shared characteristics essential for class-level understanding. In this paper, we introduce the Non-Critical Region Refinement Dataset Distillation (NRR-DD) method, which preserves instance-specific details and fine-grained regions in synthetic data while enriching non-critical regions with class-general information. This approach enables models to leverage all pixel information, capturing both feature types and enhancing overall performance. Additionally, we present Distance-Based Representative (DBR) knowledge transfer, which eliminates the need for soft labels in training by relying on the distance between synthetic data predictions and one-hot encoded labels. Experimental results show that NRR-DD achieves state-of-the-art performance on both small- and large-scale datasets. Furthermore, by storing only two distances per instance, our method delivers comparable results across various settings. The code is available at https://github.com/tmtuan1307/NRR-DD.

数据蒸馏特征平衡无软标签

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。