提出新型图像压缩框架ResiComp,提升神经编码对丢包的鲁棒性。
ResiComp: Loss-Resilient Image Compression via Dual-Functional Masked Visual Token Modeling
- 将熵建模与丢包恢复融合到统一上下文建模框架中
- 在40%丢包率下仍保持较高图像质量,且压缩效率不显著下降
- 适用于实时通信等高丢包场景,支持灵活调节效率与鲁棒性
近年来,神经图像编解码器(NICs)在压缩性能上取得显著进展,但其对传输错误的鲁棒性仍受关注不足。现有方法在实时通信中易受丢包影响。本文提出ResiComp,一种具有特征域丢包恢复(PLC)能力的神经图像压缩框架。受生成与压缩内在一致性的启发,将熵建模与丢包恢复任务统一于潜在空间上下文建模中。借鉴大语言模型中掩码视觉标记建模(MVTM)的思想,在训练时引入MVTM以模拟丢包效果,使双功能Transformer能同时预测缺失潜变量和条件概率质量函数。该方法联合优化压缩效率与抗丢包能力。此外,ResiComp支持可调编码模式,可根据网络条件灵活平衡效率与鲁棒性。大量实验表明,其在40%丢包率下仍显著提升图像质量,且压缩效率与抗丢包性之间保持合理权衡。
原文摘要 · Abstract (English)
Recent advancements in neural image codecs (NICs) are of significant compression performance, but limited attention has been paid to their error resilience. These resulting NICs tend to be sensitive to packet losses, which are prevalent in real-time communications. In this paper, we investigate how to elevate the resilience ability of NICs to combat packet losses. We propose ResiComp, a pioneering neural image compression framework with feature-domain packet loss concealment (PLC). Motivated by the inherent consistency between generation and compression, we advocate merging the tasks of entropy modeling and PLC into a unified framework focused on latent space context modeling. To this end, we take inspiration from the impressive generative capabilities of large language models (LLMs), particularly the recent advances of masked visual token modeling (MVTM). During training, we integrate MVTM to mirror the effects of packet loss, enabling a dual-functional Transformer to restore the masked latents by predicting their missing values and conditional probability mass functions. Our ResiComp jointly optimizes compression efficiency and loss resilience. Moreover, ResiComp provides flexible coding modes, allowing for explicitly adjusting the efficiency-resilience trade-off in response to varying Internet or wireless network conditions. Extensive experiments demonstrate that ResiComp can significantly enhance the NIC's resilience against packet losses, while exhibits a worthy trade-off between compression efficiency and packet loss resilience.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。