4Real-Video可生成高保真4D视频，支持多视角与时间连贯性。
4Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusion
- 采用双流架构，分别处理视角与时间维度更新。
- 相比此前方法，推理速度更快，视觉质量提升（FVD↓21.3，CLIP↑0.18）。
- 适合需要高质量动态3D场景生成的研究与应用。
我们提出4Real-Video，一种生成4D视频的新框架，以时间与视角为坐标构成视频帧网格：每行对应同一时刻的帧，每列对应同一视角的帧。设计双流架构，一路径沿列进行视角更新，另一路径沿行进行时间更新。在每个扩散变压器层后，引入同步层在两路令牌间交换信息，提供硬同步与软同步两种实现方式。该前馈架构在三方面优于先前工作：推理速度更高，视觉质量更优（FVD↓21.3，CLIP↑0.18，VideoScore↑0.15），且时间与视角一致性显著增强（VideoScore↑0.12，Dust3R-Confidence↑0.21）。
原文摘要 · Abstract (English)
We propose 4Real-Video, a novel framework for generating 4D videos, organized as a grid of video frames with both time and viewpoint axes. In this grid, each row contains frames sharing the same timestep, while each column contains frames from the same viewpoint. We propose a novel two-stream architecture. One stream performs viewpoint updates on columns, and the other stream performs temporal updates on rows. After each diffusion transformer layer, a synchronization layer exchanges information between the two token streams. We propose two implementations of the synchronization layer, using either hard or soft synchronization. This feedforward architecture improves upon previous work in three ways: higher inference speed, enhanced visual quality (measured by FVD, CLIP, and VideoScore), and improved temporal and viewpoint consistency (measured by VideoScore and Dust3R-Confidence).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。