arXiv:2512.05817cs.LG2025-12

提出统一理论框架,揭示数据蒸馏的缩放与配置覆盖规律。

Utility Boundary of Dataset Distillation: Scaling and Configuration-Coverage Laws

  • 从泛化误差视角统一分析主流数据蒸馏方法
  • 发现蒸馏样本量随规模增长呈饱和下降趋势
  • 证明所需样本数与配置多样性线性相关,指导鲁棒设计

数据蒸馏旨在构建紧凑的合成数据集,使模型在显著降低存储与计算成本的前提下,达到全数据训练的性能。尽管经验进展迅速,其理论基础仍薄弱:现有方法(梯度、分布、轨迹匹配)基于异质代理目标与优化假设,难以分析共性原则或提供通用保证。此外,当训练配置(如优化器、架构、增强策略)变化时,蒸馏数据是否仍有效尚不明确。为此,本文提出统一理论框架——配置-动态-误差分析,将主要数据蒸馏方法重新表述为同一泛化误差视角,并得出两项核心结果:(i) 缩放律,给出单配置下的误差上限,刻画误差随蒸馏样本量增加而下降的趋势,解释常见性能饱和现象;(ii) 覆盖律,表明所需蒸馏样本量与配置多样性呈线性关系,且具有可证明的上下界。此外,统一分析揭示各类匹配方法是等价的代理目标,均最小化相同泛化误差,澄清了它们为何均能实现数据蒸馏,并为代理选择如何影响样本效率与鲁棒性提供指导。跨多种方法与配置的实验验证了所推导定律,推动数据蒸馏的理论发展,支持基于理论的设计,实现紧凑且配置鲁棒的数据蒸馏。

原文摘要 · Abstract (English)

Dataset distillation (DD) aims to construct compact synthetic datasets that allow models to achieve comparable performance to full-data training while substantially reducing storage and computation. Despite rapid empirical progress, its theoretical foundations remain limited: existing methods (gradient, distribution, trajectory matching) are built on heterogeneous surrogate objectives and optimization assumptions, which makes it difficult to analyze their common principles or provide general guarantees. Moreover, it is still unclear under what conditions distilled data can retain the effectiveness of full datasets when the training configuration, such as optimizer, architecture, or augmentation, changes. To answer these questions, we propose a unified theoretical framework, termed configuration--dynamics--error analysis, which reformulates major DD approaches under a common generalization-error perspective and provides two main results: (i) a scaling law that provides a single-configuration upper bound, characterizing how the error decreases as the distilled sample size increases and explaining the commonly observed performance saturation effect; and (ii) a coverage law showing that the required distilled sample size scales linearly with configuration diversity, with provably matching upper and lower bounds. In addition, our unified analysis reveals that various matching methods are interchangeable surrogates, reducing the same generalization error, clarifying why they can all achieve dataset distillation and providing guidance on how surrogate choices affect sample efficiency and robustness. Experiments across diverse methods and configurations empirically confirm the derived laws, advancing a theoretical foundation for DD and enabling theory-driven design of compact, configuration-robust dataset distillation.

数据蒸馏理论分析缩放律配置鲁棒

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。