通过建模训练动态,大幅减少图像压缩模型的可训练参数量。
Accelerating Learned Image Compression Through Modeling Neural Training Dynamics
- 将参数分组为若干模式,用参考参数的仿射变换表示
- 训练时逐步嵌入非参考参数,参数量减少60%以上
- 适合追求高效训练的图像压缩研究者
随着学习型图像压缩(LIC)方法计算开销不断增大,提升训练效率至关重要。本文提出敏感度感知的真实与虚假嵌入训练机制(STDET),将LIC模型参数聚类为少数模式,每个模式内参数以参考参数的仿射变换表达。利用训练过程中模式内参数的稳定相关性及敏感度,逐步嵌入非参考参数,显著降低可训练参数数量。同时引入采样后移动平均(SMA)技术,通过随机梯度下降(SGD)采样权重并求移动平均,确保训练状态平滑、方差最小化。整体方法大幅降低训练空间维度和可训练参数量,且不损失模型性能,加速收敛。理论分析表明,在噪声二次模型下,该方法训练方差低于标准SGD。本方法为高效LIC训练提供了新思路。
原文摘要 · Abstract (English)
As learned image compression (LIC) methods become increasingly computationally demanding, enhancing their training efficiency is crucial. This paper takes a step forward in accelerating the training of LIC methods by modeling the neural training dynamics. We first propose a Sensitivity-aware True and Dummy Embedding Training mechanism (STDET) that clusters LIC model parameters into few separate modes where parameters are expressed as affine transformations of reference parameters within the same mode. By further utilizing the stable intra-mode correlations throughout training and parameter sensitivities, we gradually embed non-reference parameters, reducing the number of trainable parameters. Additionally, we incorporate a Sampling-then-Moving Average (SMA) technique, interpolating sampled weights from stochastic gradient descent (SGD) training to obtain the moving average weights, ensuring smooth temporal behavior and minimizing training state variances. Overall, our method significantly reduces training space dimensions and the number of trainable parameters without sacrificing model performance, thus accelerating model convergence. We also provide a theoretical analysis on the Noisy quadratic model, showing that the proposed method achieves a lower training variance than standard SGD. Our approach offers valuable insights for further developing efficient training methods for LICs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。