提出联合优化的混合精度量化框架,实现神经视频编码的高效整数化部署。
JOMP: Jointly-Optimized Mixed-Precision Quantization Across Neural Video Coding Frameworks and Buffering Strategies

- 将量化参数与位宽设为可学习变量,动态调整各模块精度
- 在保持性能接近顶尖水平的同时降低87.6%比特操作量
- 首次统一分析编码框架与缓冲策略的协同效应,适合实际部署场景
基于变分自编码器的神经视频编码在率失真性能上表现优异,但其实际应用受限于极高的计算复杂度和跨平台互操作性不足。现有方法普遍依赖浮点运算以充分发挥率失真潜力,而实际部署需整数实现。浮点转整数非易事,因编码组件间存在依赖关系,且对精度敏感度随架构变化。本文提出首个针对神经视频编码的联合优化混合精度(JOMP)框架,将量化参数与位宽作为训练过程中的可学习变量,使不同模块可运行于不同精度,从而联合优化率失真-复杂度权衡。通过系统研究不同编码框架与时间缓冲策略下的量化效果,首次建立现代编码框架与缓冲策略协同影响的统一认知。此外,构建完整整数化流水线实现确定性解码。实验表明,该框架应用于最优模型时,可实现端到端混合精度学习,达成与当前最佳方法DCVC-FM相当的率失真性能,同时减少87.6%的比特操作。
原文摘要 · Abstract (English)
Variational autoencoder-based neural video coding has demonstrated impressive rate-distortion performance. However, its adoption in real-world applications remains hindered by challenges, such as prohibitively high computational complexity and limited cross-platform interoperability. These issues are often overlooked, as most neural video codecs rely on floating-point arithmetic to fully explore their rate-distortion potential. Practical deployment, however, requires integer-based implementations. Converting floating-point implementations into integer-based networks is non-trivial, since it involves quantizing inter-dependent coding components, whose sensitivity to precision may vary across codec designs. This paper introduces a Jointly-Optimized Mixed-Precision (JOMP) framework, in which both quantization parameters and bit widths are treated as learnable variables during training. This enables different codec modules to operate at varying precision levels, thereby jointly optimizing the rate-distortion-complexity trade-off. To the best of our knowledge, JOMP is the first mixed-precision quantization framework for neural video codecs. Its effectiveness is validated through a systematic investigation of quantization across different coding frameworks and temporal buffering strategies. Our study marks the first attempt to a unified understanding of the combined effects of modern coding frameworks and temporal buffering strategies, with the aim of informing future development of neural video codecs from a practicality perspective. In addition, we develop a complete integerization pipeline to achieve deterministic decoding. Overall, when applied to our best-performing model, JOMP enables end-to-end mixed-precision learning for integer neural video codecs, achieving rate-distortion performance comparable to that of the state-of-the-art DCVC-FM while reducing bit operations by 87.6%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。