arXiv:2509.24209cs.CV2025-09被引 8

无需标定相机,快速重建动态人体4D模型并支持任意时间点生成。

Forge4D: Feed-Forward 4D Human Reconstruction and Interpolation from Uncalibrated Sparse-view Videos

  • 将4D重建拆解为3D高斯流与密集运动预测联合任务。
  • 实现任意时间点的高质量人体插值,速度优于现有方法。
  • 适合做虚拟试衣、动作捕捉等需要动态人体建模的应用。

从未标定的稀疏视角视频中实时重建动态3D人体对下游应用至关重要。现有方法或重建速度慢,或无法生成新时间帧。为此,我们提出Forge4D,一种前馈式4D人体重建与插值模型,能高效从非标定稀疏视角视频中重建时序对齐表示,支持新视角和新时间合成。模型将4D重建与插值问题转化为3D高斯流重建与密集运动预测的联合任务。在3D高斯流重建中,先从非标定稀疏图像重建静态3D高斯,再引入可学习状态令牌,在内存友好方式下跨时间戳交互更新共享信息以保证时序一致性。对于新时间合成,设计新型运动预测模块,预测相邻帧间每个3D高斯的密集运动,并结合遮挡感知的高斯融合过程,实现任意时间戳的3D高斯插值。针对密集运动监督缺乏真值的问题,将运动预测建模为密集点匹配任务,引入自监督重定向损失优化该模块;另加入遮挡感知光流损失,确保运动与合理人体运动一致,提供更强正则化。大量实验验证了模型在域内与域外数据集上的有效性。项目页与代码:https://zhenliuzju.github.io/huyingdong/Forge4D。

原文摘要 · Abstract (English)

Instant reconstruction of dynamic 3D humans from uncalibrated sparse-view videos is critical for numerous downstream applications. Existing methods, however, are either limited by the slow reconstruction speeds or incapable of generating novel-time representations. To address these challenges, we propose Forge4D, a feed-forward 4D human reconstruction and interpolation model that efficiently reconstructs temporally aligned representations from uncalibrated sparse-view videos, enabling both novel view and novel time synthesis. Our model simplifies the 4D reconstruction and interpolation problem as a joint task of streaming 3D Gaussian reconstruction and dense motion prediction. For the task of streaming 3D Gaussian reconstruction, we first reconstruct static 3D Gaussians from uncalibrated sparse-view images and then introduce learnable state tokens to enforce temporal consistency in a memory-friendly manner by interactively updating shared information across different timestamps. For novel time synthesis, we design a novel motion prediction module to predict dense motions for each 3D Gaussian between two adjacent frames, coupled with an occlusion-aware Gaussian fusion process to interpolate 3D Gaussians at arbitrary timestamps. To overcome the lack of the ground truth for dense motion supervision, we formulate dense motion prediction as a dense point matching task and introduce a self-supervised retargeting loss to optimize this module. An additional occlusion-aware optical flow loss is introduced to ensure motion consistency with plausible human movement, providing stronger regularization. Extensive experiments demonstrate the effectiveness of our model on both in-domain and out-of-domain datasets. Project page and code at: https://zhenliuzju.github.io/huyingdong/Forge4D.

4D人体重建视频生成高斯溅射运动预测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。