通过协同适配结构与语义,实现图像压缩的高效微调。
What and Where to Adapt: Structure-Semantics Co-Tuning for Machine Vision Compression via Synergistic Adapters
- 设计两种协同适配器:结构保真适配器增强特征表达,语义上下文适配器优化概率建模。
- 在四种基础编码器上达到顶尖性能,仅用少量可训练参数逼近全量微调效果。
- 适合关注高效压缩模型微调的研究者和工程应用开发者。
预训练编码器的参数高效微调是面向人眼与机器视觉图像压缩的有前景方向。现有工作多聚焦于编码器-解码器骨干网络中特征结构的调整,而对熵模型内统计语义的适应关注不足,尽管其负责预测潜在特征的概率分布。分析表明,直接将适配器插入熵模型可能导致次优结果,说明适配器的有效性高度依赖于其类型与在压缩流水线中的位置协调。为此,我们提出结构-语义协同微调(S2-CoT),通过两个专用且协同的适配器实现这一协调:结构保真适配器(SFA)嵌入编码器-解码器以动态融合空间与频率信息,保持高保真表示;语义上下文适配器(SCA)则调整熵模型,通过细化通道上下文使统计编码更高效。联合优化下,S2-CoT将潜在性能下降转化为协同增益,在四个不同基础编码器上实现当前最优结果,仅需少量可训练参数,性能接近全量微调。代码已开源:https://github.com/Brock-bit4/S2-CoT。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning of pre-trained codecs is a promising direction in image compression for human and machine vision. While most existing works have primarily focused on tuning the feature structure within the encoder-decoder backbones, the adaptation of the statistical semantics within the entropy model has received limited attention despite its function of predicting the probability distribution of latent features. Our analysis reveals that naive adapter insertion into the entropy model can lead to suboptimal outcomes, underscoring that the effectiveness of adapter-based tuning depends critically on the coordination between adapter type and placement across the compression pipeline. Therefore, we introduce Structure-Semantics Co-Tuning (S2-CoT), a novel framework that achieves this coordination via two specialized, synergistic adapters: the Structural Fidelity Adapter (SFA) and the Semantic Context Adapter (SCA). SFA is integrated into the encoder-decoder to preserve high-fidelity representations by dynamically fusing spatial and frequency information; meanwhile, the SCA adapts the entropy model to align with SFA-tuned features by refining the channel context for more efficient statistical coding. Through joint optimization, S2-CoT turns potential performance degradation into synergistic gains, achieving state-of-the-art results across four diverse base codecs with only a small fraction of trainable parameters, closely matching full fine-tuning performance. Code is available at https://github.com/Brock-bit4/S2-CoT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。