用特征与熵引导的蒸馏方法,让小模型逼近大模型压缩效果。
FEDS: Feature and Entropy-Based Distillation Strategy for Efficient Learned Image Compression
- 通过特征对齐和熵加权损失,精准传递关键信息
- 在多个数据集上仅增1.24%率失真,参数减少63%
- 适合实时或设备受限场景,通用性强
近年来,学习型图像压缩(LIC)方法在率失真性能上已超越传统编码器如VVC。然而其庞大的模型规模与高计算成本限制了实际应用。本文首先构建一个基于Swin-Transformer V2注意力模块、附加残差块和扩展潜变量通道的高容量教师模型,以提升压缩性能。在此基础上,提出一种特征与熵联合蒸馏策略(FEDS),将关键知识从教师模型迁移到轻量学生模型。具体地,通过中间特征对齐和基于熵的损失强化最具信息量的潜变量通道。采用分阶段训练方案,依次进行特征对齐、通道级蒸馏和最终微调。学生模型在Kodak(BD-Rate增加1.24%)、Tecnick(1.17%)和CLIC(0.55%)上几乎达到教师水平,同时参数量减少约63%,编码/解码速度提升约73%。消融实验表明FEDS可有效推广至基于Transformer的网络。结果证明该方法在压缩性能、速度与模型大小之间取得良好平衡,适用于实时或资源受限场景。
原文摘要 · Abstract (English)
Learned image compression (LIC) methods have recently outperformed traditional codecs such as VVC in rate-distortion performance. However, their large models and high computational costs have limited their practical adoption. In this paper, we first construct a high-capacity teacher model by integrating Swin-Transformer V2-based attention modules, additional residual blocks, and expanded latent channels, thus achieving enhanced compression performance. Building on this foundation, we propose a \underline{F}eature and \underline{E}ntropy-based \underline{D}istillation \underline{S}trategy (\textbf{FEDS}) that transfers key knowledge from the teacher to a lightweight student model. Specifically, we align intermediate feature representations and emphasize the most informative latent channels through an entropy-based loss. A staged training scheme refines this transfer in three phases: feature alignment, channel-level distillation, and final fine-tuning. Our student model nearly matches the teacher across Kodak (1.24\% BD-Rate increase), Tecnick (1.17\%), and CLIC (0.55\%) while cutting parameters by about 63\% and accelerating encoding/decoding by around 73\%. Moreover, ablation studies indicate that FEDS generalizes effectively to transformer-based networks. The experimental results demonstrate our approach strikes a compelling balance among compression performance, speed, and model parameters, making it well-suited for real-time or resource-limited scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。