arXiv:2604.11331cs.CVcs.CG2026-04被引 1

用1000个词元实现任意视角的3D场景生成,突破2D扩散模型局限。

Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale

论文配图:Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale
图 1 · 摘自论文原文
  • 构建3D隐空间编码器,将2D视觉语义转为解耦的3D表征
  • 在3D隐空间中进行扩散建模,生成空间一致且高效
  • 支持任意相机轨迹渲染,无需重复采样,适合大规模场景生成

3D场景生成长期依赖2D多视角或视频扩散模型,受限于缺乏场景级3D隐表示,且多数3D视觉数据以多视角图像或视频形式存在,天然适配2D扩散架构。此类方法将3D空间外推降维为2D时间扩展,导致:(i) 多视角表示存在显著冗余,(ii) 2D隐空间本质限制生成场景的空间一致性。本文首次提出在隐式3D隐空间中直接进行3D场景生成。首先,复用冻结的2D表示编码器构建3D表示自编码器(3DRAE),将视图关联的2D语义表示转化为视图解耦的3D隐表示,可固定复杂度、任意分辨率与画幅下表达任意视角的3D场景,语义丰富。随后引入3D扩散转换器(3DDiT),在该3D隐空间中执行扩散建模,实现高效且空间一致的3D场景生成,支持多样条件配置。此外,由于直接生成3D场景表示,可沿任意相机轨迹解码为图像和可选点云图,无需针对每条轨迹进行扩散采样,显著优于传统2D方法。

原文摘要 · Abstract (English)

3D scene generation has long been dominated by 2D multi-view or video diffusion models. This is due not only to the lack of scene-level 3D latent representation, but also to the fact that most scene-level 3D visual data exists in the form of multi-view images or videos, which are naturally compatible with 2D diffusion architectures. Typically, these 2D-based approaches degrade 3D spatial extrapolation to 2D temporal extension, which introduces two fundamental issues: (i) representing 3D scenes via 2D views leads to significant representation redundancy, and (ii) latent space rooted in 2D inherently limits the spatial consistency of the generated 3D scenes. In this paper, we propose, for the first time, to perform 3D scene generation directly within an implicit 3D latent space to address these limitations. First, we repurpose frozen 2D representation encoders to construct our 3D Representation Autoencoder (3DRAE), which grounds view-coupled 2D semantic representations into a view-decoupled 3D latent representation. This enables representing 3D scenes observed from arbitrary numbers of views--at any resolution and aspect ratio--with fixed complexity and rich semantics. Then we introduce 3D Diffusion Transformer (3DDiT), which performs diffusion modeling in this 3D latent space, achieving remarkably efficient and spatially consistent 3D scene generation while supporting diverse conditioning configurations. Moreover, since our approach directly generates a 3D scene representation, it can be decoded to images and optional point maps along arbitrary camera trajectories without requiring per-trajectory diffusion sampling pass, which is common in 2D-based approaches.

3D生成扩散模型隐空间场景生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。