用自回归框架统一神经视频压缩,降低内存占用。
UAR-NVC: A Unified AutoRegressive Framework for Memory-Efficient Neural Video Compression
- 将视频分片并行处理,结合时间序列自回归建模。
- 支持可调延迟,内存消耗显著低于传统方法。
- 适合边缘设备等资源受限场景的视频压缩应用。
隐式神经表示(INRs)在视频压缩中展现巨大潜力,但随着帧数增加,训练与推理的内存开销急剧上升,制约了其在资源受限场景的应用。受传统视频压缩逐帧处理策略启发,本文提出统一自回归框架UAR-NVC,将时间序列与INR模型统一于自回归范式下。该框架将视频划分为多个片段,每个片段使用独立的INR实例进行处理,兼顾两种压缩框架优势,并实现无缝切换。为减少片段间的时序冗余,设计了两个模块优化参数初始化、训练与压缩过程。通过调节片段长度,支持可变延迟。大量实验表明,相比基线模型,UAR-NVC在资源受限环境下显著提升压缩性能。
原文摘要 · Abstract (English)
Implicit Neural Representations (INRs) have demonstrated significant potential in video compression by representing videos as neural networks. However, as the number of frames increases, the memory consumption for training and inference increases substantially, posing challenges in resource-constrained scenarios. Inspired by the success of traditional video compression frameworks, which process video frame by frame and can efficiently compress long videos, we adopt this modeling strategy for INRs to decrease memory consumption, while aiming to unify the frameworks from the perspective of timeline-based autoregressive modeling. In this work, we present a novel understanding of INR models from an autoregressive (AR) perspective and introduce a Unified AutoRegressive Framework for memory-efficient Neural Video Compression (UAR-NVC). UAR-NVC integrates timeline-based and INR-based neural video compression under a unified autoregressive paradigm. It partitions videos into several clips and processes each clip using a different INR model instance, leveraging the advantages of both compression frameworks while allowing seamless adaptation to either in form. To further reduce temporal redundancy between clips, we design two modules to optimize the initialization, training, and compression of these model parameters. UAR-NVC supports adjustable latencies by varying the clip length. Extensive experimental results demonstrate that UAR-NVC, with its flexible video clip setting, can adapt to resource-constrained environments and significantly improve performance compared to different baseline models. The project page: "https://wj-inf.github.io/UAR-NVC-page/".
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。