arXiv:2502.20653cs.CVcs.AI2025-02CVPR被引 56

用神经特征函数最大化差异,实现高效高保真数据压缩。

Dataset Distillation with Neural Characteristic Function: A Minmax Perspective

  • 将数据蒸馏建模为极小极大问题,设计新型分布差异度量NCFD。
  • 在ImageSquawk上提升20.5%准确率,内存占用减少300倍以上。
  • 适合需要低资源高精度数据生成的场景,如边缘设备部署。

数据蒸馏已成为降低深度学习数据需求的强大方法。现有基于分布匹配的方法虽兼顾效率与性能,但常用距离度量难以准确捕捉分布差异,导致度量不可靠。本文将数据蒸馏重新建模为极小极大优化问题,提出神经特征函数差异(NCFD),一种理论完备且全面的分布差异度量。NCFD利用特征函数(CF)完整封装分布信息,通过神经网络优化CF频率参数采样策略,以最大化差异性来增强距离估计;同时最小化真实与合成数据在该度量下的差异。所提方法( mymethod{})天然对齐真实与合成数据在复平面上的相位与幅值,平衡了合成样本的真实感与多样性。实验表明,该方法在低分辨率和高分辨率数据集上均显著优于当前最优方法,尤其在ImageSquawk上取得20.5%的准确率提升。相比现有方法,内存使用减少超300倍,处理速度提高20倍。据我们所知,这是首个在单张NVIDIA 2080 Ti GPU上仅用2.3 GB内存完成CIFAR-100无损压缩的工作。

原文摘要 · Abstract (English)

Dataset distillation has emerged as a powerful approach for reducing data requirements in deep learning. Among various methods, distribution matching-based approaches stand out for their balance of computational efficiency and strong performance. However, existing distance metrics used in distribution matching often fail to accurately capture distributional differences, leading to unreliable measures of discrepancy. In this paper, we reformulate dataset distillation as a minmax optimization problem and introduce Neural Characteristic Function Discrepancy (NCFD), a comprehensive and theoretically grounded metric for measuring distributional differences. NCFD leverages the Characteristic Function (CF) to encapsulate full distributional information, employing a neural network to optimize the sampling strategy for the CF's frequency arguments, thereby maximizing the discrepancy to enhance distance estimation. Simultaneously, we minimize the difference between real and synthetic data under this optimized NCFD measure. Our approach, termed Neural Characteristic Function Matching (\mymethod{}), inherently aligns the phase and amplitude of neural features in the complex plane for both real and synthetic data, achieving a balance between realism and diversity in synthetic samples. Experiments demonstrate that our method achieves significant performance gains over state-of-the-art methods on both low- and high-resolution datasets. Notably, we achieve a 20.5\% accuracy boost on ImageSquawk. Our method also reduces GPU memory usage by over 300$\times$ and achieves 20$\times$ faster processing speeds compared to state-of-the-art methods. To the best of our knowledge, this is the first work to achieve lossless compression of CIFAR-100 on a single NVIDIA 2080 Ti GPU using only 2.3 GB of memory.

数据蒸馏分布匹配低资源训练特征函数

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。