用局部视图注意力实现大场景3D重建，单次前向传播完成
LVT: Large-Scale Scene Reconstruction via Local View Transformers
- 基于局部视图的注意力机制，避免传统Transformer的二次复杂度
- 支持任意大规模高分辨率场景重建，单次推理完成
- 适合需要高效大场景建模的研究者与工业应用
大型Transformer模型在3D视觉和新视角合成中表现出强大能力。然而，标准Transformer的二次复杂度使其难以扩展到大场景。为此，我们提出局部视图Transformer（LVT），一种无需二次注意力运算的大规模场景重建与新视角合成架构。受邻近视图比远距离视图提供更有效局部场景信息的启发，模型在每个视图周围局部邻域内处理所有信息。为捕捉邻近视图中的令牌，我们引入一种基于查询视图与邻近视图间相对几何变换的新型位置编码。模型输出解码为包含颜色与不透明度视角依赖性的3D高斯点云表示。综合来看，LVT可在单次前向传播中实现任意大、高分辨率场景的重建。项目页面含结果与交互演示：https://toobaimt.github.io/lvt/
原文摘要 · Abstract (English)
Large transformer models are proving to be a powerful tool for 3D vision and novel view synthesis. However, the standard Transformer's well-known quadratic complexity makes it difficult to scale these methods to large scenes. To address this challenge, we propose the Local View Transformer (LVT), a large-scale scene reconstruction and novel view synthesis architecture that circumvents the need for the quadratic attention operation. Motivated by the insight that spatially nearby views provide more useful signal about the local scene composition than distant views, our model processes all information in a local neighborhood around each view. To attend to tokens in nearby views, we leverage a novel positional encoding that conditions on the relative geometric transformation between the query and nearby views. We decode the output of our model into a 3D Gaussian Splat scene representation that includes both color and opacity view-dependence. Taken together, the Local View Transformer enables reconstruction of arbitrarily large, high-resolution scenes in a single forward pass. See our project page for results and interactive demos https://toobaimt.github.io/lvt/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。