arXiv:2503.01212cs.CVcs.LG2025-03中稿 · ICLR被引 3

用频谱过滤统一解释数据蒸馏,揭示其核心是匹配特定频率特征。

Understanding Dataset Distillation via Spectral Filtering

  • 将不同数据蒸馏方法统一为频谱滤波框架,通过调节特征相关矩阵的频段来实现压缩。
  • 提出渐进式频率匹配方法(CFM),可同时捕获低频全局纹理和高频局部细节。
  • 在CIFAR-10/100与ImageNet-1K上优于现有方法,适合研究高效训练与数据压缩者。

数据蒸馏(DD)作为一种压缩数据集并加速模型训练的有前景方法,其各类方法之间的内在联系仍不明确。本文提出UniDD,一个基于频谱滤波的统一框架,将不同DD目标解释为影响特征-特征相关(FFC)矩阵特征值的特定滤波函数,并调节特征-标签相关(FLC)矩阵的频率成分。该框架揭示了数据蒸馏的本质在于匹配特定频率的特征。进一步地,根据滤波行为将现有方法分为低频匹配(编码全局纹理)与高频匹配(编码局部细节)。然而,现有方法采用固定滤波函数,无法同时捕捉高低频信息。为此,本文提出课程频率匹配(CFM),逐步调整滤波参数,以覆盖FFC与FLC矩阵的低频与高频成分。在小型数据集(如CIFAR-10/100)及大规模数据集(如ImageNet-1K)上的大量实验表明,CFM显著优于现有基线,验证了UniDD的实际有效性。

原文摘要 · Abstract (English)

Dataset distillation (DD) has emerged as a promising approach to compress datasets and speed up model training. However, the underlying connections among various DD methods remain largely unexplored. In this paper, we introduce UniDD, a spectral filtering framework that unifies diverse DD objectives. UniDD interprets each DD objective as a specific filter function that affects the eigenvalues of the feature-feature correlation (FFC) matrix and modulates the frequency components of the feature-label correlation (FLC) matrix. In this way, UniDD reveals that the essence of DD fundamentally lies in matching frequency-specific features. Moreover, according to the filter behaviors, we classify existing methods into low-frequency matching and high-frequency matching, encoding global texture and local details, respectively. However, existing methods rely on fixed filter functions throughout distillation, which cannot capture the low- and high-frequency information simultaneously. To address this limitation, we further propose Curriculum Frequency Matching (CFM), which gradually adjusts the filter parameter to cover both low- and high-frequency information of the FFC and FLC matrices. Extensive experiments on small-scale datasets, such as CIFAR-10/100, and large-scale datasets, including ImageNet-1K, demonstrate the superior performance of CFM over existing baselines and validate the practicality of UniDD.

数据蒸馏频谱滤波特征匹配高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。