用低秩卷积优化视频神经编码,显著降低计算开销且保持画质。
LRConv-NeRV: Low Rank Convolution for Efficient Neural Video Compression
- 用可分解的低秩卷积替换部分密集卷积,实现端到端训练。
- 仅替换最后阶段卷积,计算量降68%,模型大小减9.3%,画质损失极小。
- 适合资源受限场景,尤其支持低精度部署和稳定时序重建。
神经视频表示(NeRV)将整个视频序列编码为神经网络参数,提供了一种不同于传统视频编解码器的新范式。然而,NeRV中的卷积解码器仍存在计算成本高、内存占用大的问题,限制了其在资源受限环境中的应用。本文提出LRConv-NeRV,通过在解码器中选择性地将部分3×3密集卷积层替换为结构化低秩可分离卷积,并在解码架构内进行端到端训练。通过从最深层向早期阶段逐步应用低秩分解,实现了重建质量与效率之间的可控权衡。大量实验表明,仅对最终解码阶段应用LRConv可使解码复杂度从201.9降至64.9 GFLOPs(降低68%),模型大小减少9.3%,同时质量损失可忽略,并实现约9.2%的码率下降。在INT8后训练量化下,LRConv-NeRV的重建质量接近全连接的NeRV基线;而对早期解码阶段进行更激进的因子分解则导致画质显著退化。相比现有工作,在层对齐设置下,LRConv-NeRV展现出更优的效率与质量平衡,大幅减少GFLOPs与参数量,同时保持更高PSNR/MS-SSIM并提升时间稳定性。基于LPIPS的时序闪烁分析进一步显示,该方法保留了接近基线的时序一致性,验证了其在低精度和资源受限场景下高效神经视频解码的潜力。
原文摘要 · Abstract (English)
Neural Representations for Videos (NeRV) encode entire video sequences within neural network parameters, offering an alternative paradigm to conventional video codecs. However, the convolutional decoder of NeRV remains computationally expensive and memory intensive, limiting its deployment in resource-constrained environments. This paper proposes LRConv-NeRV, an efficient NeRV variant that replaces selected dense 3x3 convolutional layers with structured low-rank separable convolutions, trained end-to-end within the decoder architecture. By progressively applying low-rank factorization from the largest to earlier decoder stages, LRConv-NeRV enables controllable trade-offs between reconstruction quality and efficiency. Extensive experiments demonstrate that applying LRConv only to the final decoder stage reduces decoder complexity by 68%, from 201.9 to 64.9 GFLOPs, and model size by 9.3%, while incurring negligible quality loss and achieving approximately 9.2% bitrate reduction. Under INT8 post-training quantization, LRConv-NeRV preserves reconstruction quality close to the dense NeRV baseline, whereas more aggressive factorization of early decoder stages leads to disproportionate quality degradation. Compared to existing work under layer-aligned settings, LRConv-NeRV achieves a more favorable efficiency versus quality trade-off, offering substantial GFLOPs and parameter reductions while maintaining higher PSNR/MS-SSIM and improved temporal stability. Temporal flicker analysis using LPIPS further shows that the proposed solution preserves temporal coherence close to the NeRV baseline, results establish LRConv-NeRV as a potential architectural alternative for efficient neural video decoding under low-precision and resource-constrained settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。