用分块自回归方法直接生成4K全景视频,解决VR高画质生成难题。
CubeComposer: Spatio-Temporal Autoregressive 4K 360° Video Generation from Perspective Video
- 将360°视频拆成六面立方图,按时空顺序逐步生成
- 支持原生4K分辨率输出,视觉质量优于现有方法
- 适合需要高沉浸感的VR应用开发人员使用
从透视视角输入生成高质量360°全景视频是虚拟现实(VR)的关键应用,其中高分辨率对沉浸体验尤为重要。现有方法受限于普通扩散模型的计算瓶颈,仅支持≤1K分辨率的原生生成,需依赖次优的后处理超分辨率提升。本文提出CubeComposer,一种新型时空自回归扩散模型,可原生生成4K分辨率360°视频。通过将视频分解为六个面的立方图表示,以精心规划的时空顺序自回归合成内容,在降低内存需求的同时实现高分辨率输出。具体提出:(1) 时空自回归策略,协调立方体各面与时间窗口间的生成顺序,确保连贯性;(2) 立方面上下文管理机制,采用稀疏上下文注意力设计提升效率;(3) 连续性感知技术,包括立方体感知的位置编码、填充与融合策略,消除边界接缝。在基准数据集上的大量实验表明,CubeComposer在原生分辨率和视觉质量上均优于当前最先进方法,支持实际VR应用场景。
原文摘要 · Abstract (English)
Generating high-quality 360° panoramic videos from perspective input is one of the crucial applications for virtual reality (VR), whereby high-resolution videos are especially important for immersive experience. Existing methods are constrained by computational limitations of vanilla diffusion models, only supporting $\leq$ 1K resolution native generation and relying on suboptimal post super-resolution to increase resolution. We introduce CubeComposer, a novel spatio-temporal autoregressive diffusion model that natively generates 4K-resolution 360° videos. By decomposing videos into cubemap representations with six faces, CubeComposer autoregressively synthesizes content in a well-planned spatio-temporal order, reducing memory demands while enabling high-resolution output. Specifically, to address challenges in multi-dimensional autoregression, we propose: (1) a spatio-temporal autoregressive strategy that orchestrates 360° video generation across cube faces and time windows for coherent synthesis; (2) a cube face context management mechanism, equipped with a sparse context attention design to improve efficiency; and (3) continuity-aware techniques, including cube-aware positional encoding, padding, and blending to eliminate boundary seams. Extensive experiments on benchmark datasets demonstrate that CubeComposer outperforms state-of-the-art methods in native resolution and visual quality, supporting practical VR application scenarios. Project page: https://lg-li.github.io/project/cubecomposer
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。