arXiv:2609.03931cs.CVcs.LG2026-09

用稀疏隐空间实现多视角图像的3D场景补全,无需真实3D标注。

Sparse auto-regressive modeling for scene generation from multi-view images

  • 构建稀疏体素对齐的3D隐空间,仅表示有物体的体素。
  • 在合成室内场景上生成新视角质量优于现有方法。
  • 适用于真实世界数据,适合做3D场景重建的研究者。

从稀疏、无约束视角生成完整3D场景是3D视觉中的基础挑战,需在观测内容之外进行推理,同时保持计算可处理性。现有前馈重建方法仅能还原输入图像可见内容,而3D生成模型受限于密集体素表示的高计算开销及大规模3D监督数据稀缺。我们提出SPAR3S,一种无需真实3D数据监督的稀疏体素对齐3D隐生成模型,用于条件化场景补全。核心思路是在结构化、紧凑的体素对齐3D隐空间中建模,仅表示占据体素。通过可微分3D高斯点云渲染实现光度监督,直接从多视角图像学习该稀疏隐空间。给定从稀疏输入视图编码的部分体素,场景补全转化为预测缺失隐令牌及其空间分布。为此,训练一个掩码自回归Transformer,联合建模体素占据与隐令牌值,实现高效且空间一致的未见区域生成。在合成室内场景上验证了方法有效性,新视角生成质量高于先前工作。进一步在RealEstate10k数据集上验证泛化能力,证明其适用于真实世界数据。

原文摘要 · Abstract (English)

Generating complete 3D scenes from sparse, unconstrained views is a fundamental challenge in 3D vision which requires reasoning beyond observed content while remaining computationally tractable. Existing feed-forward reconstruction methods are inherently limited to content visible in the input images, while 3D generative modeling is hindered by the high computational cost of dense volumetric representations and the scarcity of large-scale 3D supervision. We introduce SPAR3S, a sparse voxel-aligned 3D latent generative model for conditional scene completion without requiring ground-truth 3D data for supervision. Our key insight is to formulate 3D scene generation in a structured, compact, voxel-aligned 3D latent space where only occupied voxels are represented. We learn this sparse latent space directly from multi-view images using photometric supervision via differentiable 3D Gaussian Splatting. Given a partial set of observed voxels encoded from sparse input views, scene completion reduces to predicting the missing latent tokens and their spatial support within the voxel grid. To this end, we train a masked autoregressive transformer that jointly models voxel occupancy and latent token values, enabling efficient and spatially consistent generation of unseen regions. We demonstrate the effectiveness of our method on synthetic indoor scenes, achieving higher novel-view quality than prior work. We further validate its generalization on RealEstate10k, highlighting its applicability to real-world data.

3D生成稀疏建模自回归场景补全

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。