arXiv:2507.17221cs.LGcs.CV2025-07ICCV被引 3

用压缩率-效用视角优化数据蒸馏,实现更高效的数据压缩。

Dataset Distillation as Data Compression: A Rate-Utility Perspective

  • 将合成样本建模为可优化的隐码,通过轻量解码器生成。
  • 在CIFAR和ImageNet上实现最高170倍压缩,精度相当。
  • 提出新度量bpc,支持跨方法公平比较,适合研究数据压缩者。

受‘规模即一切’范式驱动,现代机器学习需要越来越大的数据集和模型,带来巨大的计算与存储开销。数据蒸馏通过将原始数据集压缩为少量合成样本,保留其完整效用,缓解此问题。但现有方法或固定存储预算下最大化性能,或追求冗余消除的合成数据表示,未联合优化两者。本文提出联合率-效用优化方法:将合成样本参数化为可优化的隐码,由极轻量网络解码;以量化隐码的香农熵作为率度量,任意现有蒸馏损失作为效用度量,通过拉格朗日乘子权衡二者。为实现跨方法公平比较,引入每类比特数(bpc),精确衡量样本、标签及解码器参数成本。在CIFAR-10、CIFAR-100和ImageNet-128上,本方法在相近精度下压缩比达标准蒸馏的170倍。在多种bpc预算、蒸馏损失和骨干网络下,始终取得更优的率-效用权衡。

原文摘要 · Abstract (English)

Driven by the ``scale-is-everything'' paradigm, modern machine learning increasingly demands ever-larger datasets and models, yielding prohibitive computational and storage requirements. Dataset distillation mitigates this by compressing an original dataset into a small set of synthetic samples, while preserving its full utility. Yet, existing methods either maximize performance under fixed storage budgets or pursue suitable synthetic data representations for redundancy removal, without jointly optimizing both objectives. In this work, we propose a joint rate-utility optimization method for dataset distillation. We parameterize synthetic samples as optimizable latent codes decoded by extremely lightweight networks. We estimate the Shannon entropy of quantized latents as the rate measure and plug any existing distillation loss as the utility measure, trading them off via a Lagrange multiplier. To enable fair, cross-method comparisons, we introduce bits per class (bpc), a precise storage metric that accounts for sample, label, and decoder parameter costs. On CIFAR-10, CIFAR-100, and ImageNet-128, our method achieves up to $170\times$ greater compression than standard distillation at comparable accuracy. Across diverse bpc budgets, distillation losses, and backbone architectures, our approach consistently establishes better rate-utility trade-offs.

数据蒸馏压缩率率-效权衡高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。