通过迭代构造扰动，让模型和数据相互优化，提升泛化性能。
Self Distillation via Iterative Constructive Perturbations
- 用模型损失引导输入数据迭代改进，生成更优表示
- 交替优化模型参数与输入数据，缓解过拟合与泛化差距
- 适用于多种训练场景，显著提升模型性能
深度神经网络在多个领域取得显著成果，但训练时性能与泛化能力的平衡仍是挑战。本文提出一种新框架，采用循环优化策略，同时优化模型及其输入数据，重新思考传统训练范式。核心是迭代构造扰动（ICP），利用模型损失迭代扰动输入，在若干优化步骤中逐步构建增强表示。该ICP输入反馈至模型，生成改进的中间特征，作为自蒸馏框架中与原始特征对比的目标。通过交替调整模型参数与数据，方法有效缩小拟合与泛化之间的差距，带来性能提升。大量实验表明，该方法不仅缓解了神经网络常见的性能瓶颈，还在多种训练变体中展现出显著优势。
原文摘要 · Abstract (English)
Deep Neural Networks have achieved remarkable achievements across various domains, however balancing performance and generalization still remains a challenge while training these networks. In this paper, we propose a novel framework that uses a cyclic optimization strategy to concurrently optimize the model and its input data for better training, rethinking the traditional training paradigm. Central to our approach is Iterative Constructive Perturbation (ICP), which leverages the model's loss to iteratively perturb the input, progressively constructing an enhanced representation over some refinement steps. This ICP input is then fed back into the model to produce improved intermediate features, which serve as a target in a self-distillation framework against the original features. By alternately altering the model's parameters to the data and the data to the model, our method effectively addresses the gap between fitting and generalization, leading to enhanced performance. Extensive experiments demonstrate that our approach not only mitigates common performance bottlenecks in neural networks but also demonstrates significant improvements across training variations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。