用频谱过滤统一解释数据蒸馏,揭示其核心是匹配特定频率特征。
Understanding Dataset Distillation via Spectral Filtering
- 将不同数据蒸馏方法统一为频谱滤波框架,通过调节特征相关矩阵的频段来实现压缩。
- 提出渐进式频率匹配方法(CFM),可同时捕获低频全局纹理和高频局部细节。
- 在CIFAR-10/100与ImageNet-1K上优于现有方法,适合研究高效训练与数据压缩者。
数据蒸馏(DD)作为一种压缩数据集并加速模型训练的有前景方法,其各类方法之间的内在联系仍不明确。本文提出UniDD,一个基于频谱滤波的统一框架,将不同DD目标解释为影响特征-特征相关(FFC)矩阵特征值的特定滤波函数,并调节特征-标签相关(FLC)矩阵的频率成分。该框架揭示了数据蒸馏的本质在于匹配特定频率的特征。进一步地,根据滤波行为将现有方法分为低频匹配(编码全局纹理)与高频匹配(编码局部细节)。然而,现有方法采用固定滤波函数,无法同时捕捉高低频信息。为此,本文提出课程频率匹配(CFM),逐步调整滤波参数,以覆盖FFC与FLC矩阵的低频与高频成分。在小型数据集(如CIFAR-10/100)及大规模数据集(如ImageNet-1K)上的大量实验表明,CFM显著优于现有基线,验证了UniDD的实际有效性。
原文摘要 · Abstract (English)
Dataset distillation (DD) has emerged as a promising approach to compress datasets and speed up model training. However, the underlying connections among various DD methods remain largely unexplored. In this paper, we introduce UniDD, a spectral filtering framework that unifies diverse DD objectives. UniDD interprets each DD objective as a specific filter function that affects the eigenvalues of the feature-feature correlation (FFC) matrix and modulates the frequency components of the feature-label correlation (FLC) matrix. In this way, UniDD reveals that the essence of DD fundamentally lies in matching frequency-specific features. Moreover, according to the filter behaviors, we classify existing methods into low-frequency matching and high-frequency matching, encoding global texture and local details, respectively. However, existing methods rely on fixed filter functions throughout distillation, which cannot capture the low- and high-frequency information simultaneously. To address this limitation, we further propose Curriculum Frequency Matching (CFM), which gradually adjusts the filter parameter to cover both low- and high-frequency information of the FFC and FLC matrices. Extensive experiments on small-scale datasets, such as CIFAR-10/100, and large-scale datasets, including ImageNet-1K, demonstrate the superior performance of CFM over existing baselines and validate the practicality of UniDD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。