用条件互信息优化数据蒸馏，让小数据集更易学且效果更好
Going Beyond Feature Similarity: Effective Dataset Distillation based on Class-Aware Conditional Mutual Information
- 引入条件互信息衡量数据复杂度，指导合成数据生成
- 在多个数据集上提升蒸馏性能，训练效率更高
- 适合需要高效训练的场景，尤其对压缩数据有要求的研究者
数据蒸馏（DD）旨在通过构建小型合成数据集，降低深度神经网络在大规模数据集上训练的时间和内存开销，同时保持与原始数据相当的性能。然而，现有方法常因过度依赖特征相似性度量（如分布匹配）压缩信息，导致合成数据过于难学。本文提出基于类感知条件互信息（CMI）的新方法，通过最小化预训练网络特征空间中的经验CMI，同时约束合成数据的类相关复杂度。大量实验表明，该方法可作为通用正则化策略，显著提升现有蒸馏方法的性能与训练效率。
原文摘要 · Abstract (English)
Dataset distillation (DD) aims to minimize the time and memory consumption needed for training deep neural networks on large datasets, by creating a smaller synthetic dataset that has similar performance to that of the full real dataset. However, current dataset distillation methods often result in synthetic datasets that are excessively difficult for networks to learn from, due to the compression of a substantial amount of information from the original data through metrics measuring feature similarity, e,g., distribution matching (DM). In this work, we introduce conditional mutual information (CMI) to assess the class-aware complexity of a dataset and propose a novel method by minimizing CMI. Specifically, we minimize the distillation loss while constraining the class-aware complexity of the synthetic dataset by minimizing its empirical CMI from the feature space of pre-trained networks, simultaneously. Conducting on a thorough set of experiments, we show that our method can serve as a general regularization method to existing DD methods and improve the performance and training efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。