arXiv:2510.04838cs.CVcs.LG2025-10NeurIPS被引 7

动态调整训练截断位置,让数据蒸馏更高效准确。

Beyond Random: Automatic Inner-loop Optimization in Dataset Distillation

  • 根据梯度行为自动选择截断点和窗口大小
  • 在多个数据集上提升准确率平均6.16%
  • 加速内循环优化3.9倍,节省63%内存

深度学习对效率的需求推动了数据蒸馏技术的发展,该技术可在保持模型性能的同时压缩训练数据集。然而,现有数据蒸馏的内循环优化方法通常依赖随机截断策略,灵活性差且结果常不理想。本文观察到神经网络在不同训练阶段(早期、中期、晚期)的学习动态存在差异,导致随机截断效果不佳。为此,提出自动截断反向传播通过时间(AT-BPTT)框架,动态适应截断位置与窗口大小。AT-BPTT包含三个核心组件:(1) 基于阶段感知的时间步概率选择机制;(2) 根据梯度变化自适应调整窗口大小;(3) 低秩海塞近似以降低计算开销。在CIFAR-10、CIFAR-100、Tiny-ImageNet和ImageNet-1K上的大量实验表明,AT-BPTT达到当前最优性能,平均比基线方法提升6.16%准确率。同时,内循环优化速度提升3.9倍,内存消耗减少63%。

原文摘要 · Abstract (English)

The growing demand for efficient deep learning has positioned dataset distillation as a pivotal technique for compressing training dataset while preserving model performance. However, existing inner-loop optimization methods for dataset distillation typically rely on random truncation strategies, which lack flexibility and often yield suboptimal results. In this work, we observe that neural networks exhibit distinct learning dynamics across different training stages-early, middle, and late-making random truncation ineffective. To address this limitation, we propose Automatic Truncated Backpropagation Through Time (AT-BPTT), a novel framework that dynamically adapts both truncation positions and window sizes according to intrinsic gradient behavior. AT-BPTT introduces three key components: (1) a probabilistic mechanism for stage-aware timestep selection, (2) an adaptive window sizing strategy based on gradient variation, and (3) a low-rank Hessian approximation to reduce computational overhead. Extensive experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-1K show that AT-BPTT achieves state-of-the-art performance, improving accuracy by an average of 6.16% over baseline methods. Moreover, our approach accelerates inner-loop optimization by 3.9x while saving 63% memory cost.

数据蒸馏优化算法高效训练神经网络

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。