arXiv:2503.15855cs.CVcs.AI2025-03ICCV被引 14

用视频模型直接生成带灵活视角的3D高斯点云,无需后处理优化。

VideoRFSplat: Direct Scene-Level Text-to-3D Gaussian Splatting Generation with Flexible Pose and Multi-View Joint Modeling

  • 双流架构分离图像与相机位姿生成,减少模态干扰。
  • 异步采样使位姿更快去噪,提升跨模态一致性。
  • 在4个真实场景数据集上超越现有方法,无需后处理。

我们提出VideoRFSplat,一种直接基于文本生成3D高斯点云(3DGS)的模型,利用视频生成模型实现对无界真实场景的逼真重建。为生成多样化的相机位姿和无界空间范围,同时保证对任意文本提示的泛化能力,现有方法通常微调2D生成模型以联合建模相机位姿与多视角图像。但此类方法因模态差距导致联合建模不稳定,需额外模型稳定训练与推理。本文提出一种新架构与采样策略,在微调视频生成模型时联合建模多视角图像与相机位姿。核心思想是双流结构:通过通信模块将专用位姿生成模型连接至预训练视频生成模型,分别生成多视角图像与相机位姿,降低模态间干扰。此外,提出异步采样策略,使相机位姿比多视角图像更快去噪,快速去噪的位姿作为条件引导图像生成,减少相互模糊,增强跨模态一致性。在多个大规模真实场景数据集(RealEstate10K、MVImgNet、DL3DV-10K、ACID)上训练,VideoRFSplat优于依赖分数蒸馏采样的现有直接生成方法,且无需后处理优化。

原文摘要 · Abstract (English)

We propose VideoRFSplat, a direct text-to-3D model leveraging a video generation model to generate realistic 3D Gaussian Splatting (3DGS) for unbounded real-world scenes. To generate diverse camera poses and unbounded spatial extent of real-world scenes, while ensuring generalization to arbitrary text prompts, previous methods fine-tune 2D generative models to jointly model camera poses and multi-view images. However, these methods suffer from instability when extending 2D generative models to joint modeling due to the modality gap, which necessitates additional models to stabilize training and inference. In this work, we propose an architecture and a sampling strategy to jointly model multi-view images and camera poses when fine-tuning a video generation model. Our core idea is a dual-stream architecture that attaches a dedicated pose generation model alongside a pre-trained video generation model via communication blocks, generating multi-view images and camera poses through separate streams. This design reduces interference between the pose and image modalities. Additionally, we propose an asynchronous sampling strategy that denoises camera poses faster than multi-view images, allowing rapidly denoised poses to condition multi-view generation, reducing mutual ambiguity and enhancing cross-modal consistency. Trained on multiple large-scale real-world datasets (RealEstate10K, MVImgNet, DL3DV-10K, ACID), VideoRFSplat outperforms existing text-to-3D direct generation methods that heavily depend on post-hoc refinement via score distillation sampling, achieving superior results without such refinement.

3D生成高斯点云视频生成文本到3D

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。