解决视频神经表示的高频细节丢失问题,提升画质与运动表现。
SNeRV: Spectra-preserving Neural Representation for Video
- 用小波变换分离视频的高低频特征,针对性处理频谱偏差。
- 仅编码低频信息,高频纹理由解码器重建,兼顾压缩与细节。
- 新增时空频率分解模块,更好捕捉帧间动态变化,适合高质量视频重建。
神经视频表示(NeRV)通过神经网络参数化视频信号,开辟了视频表征的新路径。然而,现有方法受频谱偏差影响,难以有效捕捉细粒度空间细节和运动模式,即神经网络学习高频成分的速度远慢于低频成分。本文提出谱保真神经视频表示(SNeRV),通过2D离散小波变换(DWT)将视频分解为低频(LF)和高频(HF)特征,保留空间结构并直接缓解频谱偏差。为平衡紧凑性,仅编码低频成分,而包含精细纹理的高频成分由解码器生成。引入多分辨率融合单元(MFU)与高频恢复模块(HFR)增强主干网络表征能力。进一步地,将方法扩展至时域,通过额外频率分解建模相邻帧间的时序相关性,利用时序扩展上采样块(TUBs)嵌入时空低频特征。实验表明,SNeRV在还原细粒度细节方面优于现有NeRV模型,显著提升重建质量,是隐式视频表示领域的重要进展。代码已开源:https://github.com/qwertja/SNeRV。
原文摘要 · Abstract (English)
Neural representation for video (NeRV), which employs a neural network to parameterize video signals, introduces a novel methodology in video representations. However, existing NeRV-based methods have difficulty in capturing fine spatial details and motion patterns due to spectral bias, in which a neural network learns high-frequency (HF) components at a slower rate than low-frequency (LF) components. In this paper, we propose spectra-preserving NeRV (SNeRV) as a novel approach to enhance implicit video representations by efficiently handling various frequency components. SNeRV uses 2D discrete wavelet transform (DWT) to decompose video into LF and HF features, preserving spatial structures and directly addressing the spectral bias issue. To balance the compactness, we encode only the LF components, while HF components that include fine textures are generated by a decoder. Specialized modules, including a multi-resolution fusion unit (MFU) and a high-frequency restorer (HFR), are integrated into a backbone to facilitate the representation. Furthermore, we extend SNeRV to effectively capture temporal correlations between adjacent video frames, by casting the extension as additional frequency decomposition to a temporal domain. This approach allows us to embed spatio-temporal LF features into the network, using temporally extended up-sampling blocks (TUBs). Experimental results demonstrate that SNeRV outperforms existing NeRV models in capturing fine details and achieves enhanced reconstruction, making it a promising approach in the field of implicit video representations. The codes are available at https://github.com/qwertja/SNeRV.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。