用时间一致性提升视频神经表示压缩效率,实现更快更小的高清视频编码。
TeCoNeRV: Leveraging Temporal Coherence for Compressible Neural Representations for Videos
- 分段处理视频块并按时空分解权重预测,降低预训练内存开销20倍
- 仅存储相邻片段间的差异,大幅减少码流大小
- 通过时间一致性正则化,让权重变化与视频内容同步,提升压缩质量
隐式神经表示(INRs)在视频压缩中表现优异,但需为每段视频单独过拟合,难以高效处理高分辨率视频。基于超网络的方法虽可快速预测未见视频的INR权重,却存在质量低、压缩后体积大、高分辨率下内存需求过高的问题。本文提出TeCoNeRV,通过三项关键改进:(1)将权重预测任务按空间与时间分解,将短视频段划分为块管(patch tubelets),使预训练内存开销降低20倍;(2)采用基于残差的存储方案,仅保存连续片段表示间的差异,显著减少码流大小;(3)引入时间一致性正则化框架,促使权重空间变化与视频内容变化相关联。实验表明,该方法在UVG数据集上于480p和720p下分别比基线提升2.47dB和5.35dB PSNR,比特率降低36%,编码速度提升1.5-3倍。首次实现超网络方法在UVG、HEVC和MCL-JCV数据集上480p、720p及1080p的完整结果。
原文摘要 · Abstract (English)
Implicit Neural Representations (INRs) have recently demonstrated impressive performance for video compression. However, since a separate INR must be overfit for each video, scaling to high-resolution videos while maintaining encoding efficiency remains a significant challenge. Hypernetwork-based approaches predict INR weights (hyponetworks) for unseen videos at high speeds, but with low quality, large compressed size, and prohibitive memory needs at higher resolutions. We address these fundamental limitations through three key contributions: (1) an approach that decomposes the weight prediction task spatially and temporally, by breaking short video segments into patch tubelets, to reduce the pretraining memory overhead by 20$\times$; (2) a residual-based storage scheme that captures only differences between consecutive segment representations, significantly reducing bitstream size; and (3) a temporal coherence regularization framework that encourages changes in the weight space to be correlated with video content. Our proposed method, TeCoNeRV, achieves substantial improvements of 2.47dB and 5.35dB PSNR over the baseline at 480p and 720p on UVG, with 36% lower bitrates and 1.5-3$\times$ faster encoding speeds. With our low memory usage, we are the first hypernetwork approach to demonstrate results at 480p, 720p and 1080p on UVG, HEVC and MCL-JCV. Our project page is available at https://namithap10.github.io/teconerv/ .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。