arXiv:2412.03552cs.CV2024-12NeurIPS被引 27

将普通视频转为沉浸式360度视频,支持丰富动态与高质量生成。

Imagine360: Immersive 360 Video Generation from Perspective Anchor

  • 通过双分支结构融合局部视角与全局全景约束,提升生成一致性。
  • 引入反向像素掩码捕捉跨半球长程运动依赖,增强镜头反转效果。
  • 适配不同俯仰角变化,支持多样输入视频的自适应处理。

360°视频提供全方位动态场景的沉浸式体验。为实现更易用、个性化的360°视频内容创作,本文提出Imagine360,首个从标准视角视频生成高质量360°等距圆柱投影视频的框架。该方法基于少量360°视频数据,学习精细的球面视觉与运动模式。首先采用双分支设计,包含视角与全景去噪分支,分别提供局部与全局约束,并通过在扩展网络360°视频上微调运动模块与空间LoRA层。其次,设计反向像素掩码以捕捉跨半球的长程运动依赖,增强对极像素间的反向相机运动建模。第三,提出俯仰感知设计,适应因帧间俯仰变化导致的视频遮挡差异。大量实验表明,Imagine360在图像质量与运动连贯性方面优于当前主流360°视频生成方法。我们相信其可推动个性化沉浸式视频创作的发展。

原文摘要 · Abstract (English)

$360^\circ$ videos offer a hyper-immersive experience that allows the viewers to explore a dynamic scene from full 360 degrees. To achieve more user-friendly and personalized content creation in $360^\circ$ video format, we seek to lift standard perspective videos into $360^\circ$ equirectangular videos. To this end, we introduce Imagine360, the first perspective-to-$360^\circ$ video generation framework that creates high-quality $360^\circ$ videos with rich and diverse motion patterns from video anchors. Imagine360 learns fine-grained spherical visual and motion patterns from limited $360^\circ$ video data with several key designs. 1) Firstly we adopt the dual-branch design, including a perspective and a panorama video denoising branch to provide local and global constraints for $360^\circ$ video generation, with motion module and spatial LoRA layers fine-tuned on extended web $360^\circ$ videos. 2) Additionally, an antipodal mask is devised to capture long-range motion dependencies, enhancing the reversed camera motion between antipodal pixels across hemispheres. 3) To handle diverse perspective video inputs, we propose elevation-aware designs that adapt to varying video masking due to changing elevations across frames. Extensive experiments show Imagine360 achieves superior graphics quality and motion coherence among state-of-the-art $360^\circ$ video generation methods. We believe Imagine360 holds promise for advancing personalized, immersive $360^\circ$ video creation.

360视频视频生成扩散模型沉浸式

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。