文本生成360度动态全景场景,支持任意视角沉浸体验
TiP4GEN: Text to Immersive Panorama 4D Scene Generation

- 双分支生成模型协同处理全景与局部视角内容
- 融合3D高斯泼溅实现几何一致且时序连贯的动态重建
- 适合虚拟现实、元宇宙内容创作及交互式场景生成
随着VR/AR技术的快速发展和普及,高质量沉浸式动态场景的需求日益增长。然而,现有方法主要聚焦静态场景或有限视角的动态场景,难以提供任意视角下的360度沉浸体验。本文提出TiP4GEN,一种先进的文本到动态全景场景生成框架,可实现细粒度内容控制,并合成运动丰富、几何一致的全景4D场景。该框架结合全景视频生成与动态场景重建:在视频生成方面,采用双分支模型(全景分支与视角分支),通过双向交叉注意力实现跨分支信息充分交互;在场景重建方面,基于3D高斯泼溅设计几何对齐重建模型,利用度量深度图对齐时空点云,并以估计姿态初始化场景相机,保障场景的几何一致性与时序连贯性。大量实验验证了方法设计的有效性及在生成视觉逼真、运动协调的动态全景场景上的优越表现。项目主页见https://ke-xing.github.io/TiP4GEN/
原文摘要 · Abstract (English)
With the rapid advancement and widespread adoption of VR/AR technologies, there is a growing demand for the creation of high-quality, immersive dynamic scenes. However, existing generation works predominantly concentrate on the creation of static scenes or narrow perspective-view dynamic scenes, falling short of delivering a truly 360-degree immersive experience from any viewpoint. In this paper, we introduce \textbf{TiP4GEN}, an advanced text-to-dynamic panorama scene generation framework that enables fine-grained content control and synthesizes motion-rich, geometry-consistent panoramic 4D scenes. TiP4GEN integrates panorama video generation and dynamic scene reconstruction to create 360-degree immersive virtual environments. For video generation, we introduce a \textbf{Dual-branch Generation Model} consisting of a panorama branch and a perspective branch, responsible for global and local view generation, respectively. A bidirectional cross-attention mechanism facilitates comprehensive information exchange between the branches. For scene reconstruction, we propose a \textbf{Geometry-aligned Reconstruction Model} based on 3D Gaussian Splatting. By aligning spatial-temporal point clouds using metric depth maps and initializing scene cameras with estimated poses, our method ensures geometric consistency and temporal coherence for the reconstructed scenes. Extensive experiments demonstrate the effectiveness of our proposed designs and the superiority of TiP4GEN in generating visually compelling and motion-coherent dynamic panoramic scenes. Our project page is at https://ke-xing.github.io/TiP4GEN/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。