解决数据无感知识蒸馏中生成图像质量不稳定问题
Close Shortcut Wins Long: Seeking Diverse and Stable Generators for Data-Free Knowledge Distillation
- 从频域角度增强生成器对全频谱的关注
- 提出跨阶段频域重建任务提升训练稳定性
- 适合关注生成质量与模型鲁棒性的研究者
数据无感知识蒸馏（DFKD）在不访问真实数据的前提下实现知识迁移，保护隐私。然而，现有基于生成器的DFKD方法存在过度依赖教师偏好和模式崩溃问题，表现为频域中的‘生成捷径学习’：仅关注特定频率成分和位置，导致合成图像质量不一致、类别多样性差。本文提出CSWL框架，从频域视角提升生成器多样性与训练稳定性，以长期关闭捷径学习现象。为解决生成捷径学习问题，我们在特征层面引入频域增强，促使生成器关注完整频谱，抑制捷径行为。为应对训练不稳定性，提出跨阶段频域重建（CSFR）辅助任务，隐式构建指数移动平均（EMA）机制，促进长期优化与稳定。大量实验涵盖下游任务及多种分辨率下的图像识别数据集，验证了CSWL在频域视角下提升多样性和稳定性的有效性。
原文摘要 · Abstract (English)
Data-Free Knowledge Distillation (DFKD) preserves privacy by transferring knowledge without real data access. However, existing generator-based DFKD methods suffer from over-reliance on teacher preferences and pattern collapse, exhibiting "generative shortcut learning" in the frequency domain: dependent on specific frequency components and frequency positions, resulting in inconsistent synthetic image quality and class diversity. In this paper, we propose a CSWL framework aimed at introducing insights from the frequency domain perspective to improve generator diversity and training stability to Close the phenomenon of Shortcut learning to Win in the Longer term. To address the issue of generative shortcut learning, we introduce frequency-domain augmentation at the feature level, encouraging the generator to attend to the full frequency spectrum and thereby suppress shortcut learning behavior. To tackle training instability, we propose a Cross-Stage Frequency Reconstruction (CSFR) auxiliary task, which implicitly constructs an Exponential Moving Average (EMA) mechanism to promote long-term optimization and stability. Extensive experiments, including downstream tasks and various image recognition datasets at multiple resolutions, validate the effectiveness of CSWL in improving both diversity and stability from the frequency view.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。