用Transformer统一3D重建与生成，从部分视角还原完整三维结构。
RnG: A Unified Transformer for Complete 3D Modeling from Partial Observations
- 通过注意力机制分离重建与生成，用缓存隐式存储完整3D信息
- 在4个基准数据集上实现最优重建与新视角生成效果
- 支持实时交互，适合需要完整3D建模的虚拟现实应用
人类通过有限视角的二维观察感知三维世界。尽管当前前馈通用3D重建模型能从稀疏图像中恢复三维结构，其表示通常局限于观测区域，未见几何无法建模。这引发一个根本性挑战：能否从部分二维观测中推断出完整的三维结构？我们提出RnG（Reconstruction and Generation），一种新颖的前馈Transformer，通过预测隐式完整的3D表示，统一重建与生成任务。RnG的核心是重建引导的因果注意力机制，它在注意力层面分离重建与生成，并将键值缓存（KV-cache）视为隐式3D表示。任意姿态可高效查询该缓存，生成高保真新视角RGBD输出。结果表明，RnG不仅准确重建可见几何，还能生成合理且连贯的不可见几何与外观。该方法在四个基准数据集上均达到最先进的通用3D重建与新视角生成性能，同时具备实时交互能力。
原文摘要 · Abstract (English)
Human perceive the 3D world through 2D observations from limited viewpoints. While recent feed-forward generalizable 3D reconstruction models excel at recovering 3D structures from sparse images, their representations are often confined to observed regions, leaving unseen geometry un-modeled. This raises a key, fundamental challenge: Can we infer a complete 3D structure from partial 2D observations? We present RnG (Reconstruction and Generation), a novel feed-forward Transformer that unifies these two tasks by predicting an implicit, complete 3D representation. At the core of RnG, we propose a reconstruction-guided causal attention mechanism that separates reconstruction and generation at the attention level, and treats the KV-cache as an implicit 3D representation. Then, arbitrary poses can efficiently query this cache to render high-fidelity, novel-view RGBD outputs. As a result, RnG not only accurately reconstructs visible geometry but also generates plausible, coherent unseen geometry and appearance. Our method achieves state-of-the-art performance in both generalizable 3D reconstruction and novel view generation, while operating efficiently enough for real-time interactive applications. Project page: https://npucvr.github.io/RnG
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。