提出4DSTAR模型,实现时空一致的4D物体自回归生成。
Spatial-Temporal State Propagation Autoregressive Model for 4D Object Generation
- 分时步分组预测,通过动态状态传播建模长时依赖。
- 在ScanNet和ShapeNet上生成结果时空一致性显著提升。
- 适合需要高保真动态3D生成的科研与工业场景。
生成具有时空一致性的高质量4D物体仍具挑战。现有基于扩散的方法常因未能利用所有历史时刻输出引导当前时刻生成,导致时空不一致。为此,我们提出空间-时间状态传播自回归模型(4DSTAR),可生成保持时空一致性的4D物体。4DSTAR将生成问题建模为对表示4D物体的标记(tokens)的预测。其包含两个关键组件:(1) 动态时空状态传播自回归模型(STAR),通过将预测标记按时间步分组,利用历史组的状态传播建模长期依赖,并通过一个动态更新的时空容器,聚合所有历史组的有效状态特征,作为条件特征指导下一组标记的预测;(2) 4D VQ-VAE,将4D结构隐式编码至离散空间,并将STAR预测的离散标记解码为时序连贯的动态3D高斯。实验表明,4DSTAR能生成时空一致的4D物体,在性能上可媲美扩散模型。
原文摘要 · Abstract (English)
Generating high-quality 4D objects with spatial-temporal consistency is still formidable. Existing diffusion-based methods often struggle with spatial-temporal inconsistency, as they fail to leverage outputs from all previous timesteps to guide the generation at the current timestep. Therefore, we propose a Spatial-Temporal State Propagation AutoRegressive Model (4DSTAR), which generates 4D objects maintaining temporal-spatial consistency. 4DSTAR formulates the generation problem as the prediction of tokens that represent the 4D object. It consists of two key components: (1) The dynamic spatial-temporal state propagation autoregressive model (STAR) is proposed, which achieves spatial-temporal consistent generation. Unlike standard autoregressive models, STAR divides prediction tokens into groups based on timesteps. It models long-term dependencies by propagating spatial-temporal states from previous groups and utilizes these dependencies to guide generation at the next timestep. To this end, a spatial-temporal container is proposed, which dynamically updating the effective spatial-temporal state features from all historical groups, then updated features serve as conditional features to guide the prediction of the next token group. (2) The 4D VQ-VAE is proposed, which implicitly encodes the 4D structure into discrete space and decodes the discrete tokens predicted by STAR into temporally coherent dynamic 3D Gaussians. Experiments demonstrate that 4DSTAR generates spatial-temporal consistent 4D objects, and achieves performance competitive with diffusion models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。