通过分阶段蒸馏，让小型图像压缩模型更快更稳地训练。
Efficient training for compact compression models via sequential distillation
- 先用简化版师生结构分步蒸馏，再逐步增加复杂度。
- 轻量模型在初期就保持高质量重建和统计特性。
- 适合算力有限的设备部署，尤其对小模型友好。
面向硬件受限场景的图像压缩深度学习模型常因结构复杂、训练成本高而难以应用。本文提出一种基于顺序蒸馏的高效训练方法，显著压缩基于自编码器的压缩网络。核心思路是：初期使用简化优化目标帮助极简架构快速收敛，随后逐步引入复杂性。首先进行编码器-解码器的分阶段蒸馏，为轻量模型提供稳健初始化；之后采用标准训练并辅以潜在空间蒸馏进行正则化。在两种不同架构上评估轻量自编码器，实验表明，该方法在训练早期即能更好保持重建质量与统计保真度，优于直接使用原始损失训练轻量模型，显著提升资源受限环境下的实用性。
原文摘要 · Abstract (English)
Deep learning models for image compression often face practical limitations in hardware-constrained applications. Although these models achieve high-quality reconstructions, they are typically complex, heavyweight, and require substantial training data and computational resources. We propose a methodology to significantly reduce autoencoder-based compression networks in a more stable Knowledge Distillation process. The intuition is that highly reduced architectures benefit from simplified optimization objectives in early training, with complexity gradually introduced later. Therefore, our approach begins with a sequential encoder--decoder distillation stage that provides a robust initialization for the lightweight model. This is followed by standard training that can be regularized with latent distillation. We evaluate the resulting lightweight autoencoders across two different architectures on the image compression task. Experiments show that our method preserves reconstruction quality and statistical fidelity in early epochs better than training lightweight autoencoders with the original loss, making it practical for resource-limited environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。