arXiv:2501.02427cs.CV2025-01AAAI被引 10

MetaNeRV让视频神经表示快速适配新视频,训练效率提升显著。

MetaNeRV: Meta Neural Representations for Videos with Spatial-Temporal Guidance

  • 用元学习初始化参数,快速适配新视频
  • 多分辨率空间引导+渐进式时间引导,提升表示能力
  • 适合需要快速视频建模的场景,如压缩与分析

视频的神经表示(NeRV)作为新兴的隐式神经表示方法,将视频视为以帧索引为输入的神经网络。然而,现有方法在面对大量多样视频时适应效率低,因每段视频需从头训练独立模型。同时,其空间上需从低维时间戳生成高维图像信号,且视频通常包含数十帧,相邻帧间变化微小。为提升视频表示效率,本文提出面向视频的元神经表示框架MetaNeRV。该框架利用元学习学习最优参数初始化,作为新视频快速适应的良好起点。针对视频特有的时空特性,引入时空引导机制:空间引导通过多分辨率损失捕捉不同层级信息,时间引导采用渐进学习策略在元训练过程中逐步细化拟合帧数。在多个数据集上的实验表明,MetaNeRV在视频表示与视频压缩任务中均表现更优。

原文摘要 · Abstract (English)

Neural Representations for Videos (NeRV) has emerged as a promising implicit neural representation (INR) approach for video analysis, which represents videos as neural networks with frame indexes as inputs. However, NeRV-based methods are time-consuming when adapting to a large number of diverse videos, as each video requires a separate NeRV model to be trained from scratch. In addition, NeRV-based methods spatially require generating a high-dimension signal (i.e., an entire image) from the input of a low-dimension timestamp, and a video typically consists of tens of frames temporally that have a minor change between adjacent frames. To improve the efficiency of video representation, we propose Meta Neural Representations for Videos, named MetaNeRV, a novel framework for fast NeRV representation for unseen videos. MetaNeRV leverages a meta-learning framework to learn an optimal parameter initialization, which serves as a good starting point for adapting to new videos. To address the unique spatial and temporal characteristics of video modality, we further introduce spatial-temporal guidance to improve the representation capabilities of MetaNeRV. Specifically, the spatial guidance with a multi-resolution loss aims to capture the information from different resolution stages, and the temporal guidance with an effective progressive learning strategy could gradually refine the number of fitted frames during the meta-learning process. Extensive experiments conducted on multiple datasets demonstrate the superiority of MetaNeRV for video representations and video compression.

视频生成元学习神经表示

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。