扩大上下文窗口提升3D物体重建质量,实现高保真逆渲染。
LSRM: High-Fidelity Object-Centric Reconstruction via Scaled Context Windows
- 通过稀疏注意力与分层优化,聚焦关键区域提升计算效率。
- 相比现有方法,对象令牌多20倍,图像令牌多2倍,重建更精细。
- 适合需要高细节3D重建与逆渲染的研究者和工业应用。
我们提出大型稀疏重建模型(LSRM),研究扩展Transformer上下文窗口对前馈式3D重建的影响。尽管近期的物体中心前馈方法已能生成高质量重建结果,但在恢复细微纹理和外观方面仍落后于密集视图优化方法。本文表明,通过大幅增加活跃物体和图像令牌数量来扩展上下文窗口,可显著缩小这一差距,实现高保真3D物体重建与逆渲染。为高效扩展,我们针对3D重建改进了原生稀疏注意力机制,提出三项关键贡献:(1) 一种高效的粗到精流水线,通过预测稀疏高分辨率残差聚焦信息区域;(2) 一种基于显式几何距离而非标准注意力分数的3D感知空间路由机制,建立准确的2D-3D对应关系;(3) 一种块感知的序列并行策略,配合All-gather-KV协议,平衡跨GPU的动态稀疏负载。实验表明,LSRM处理的物体令牌比之前最先进方法多20倍,图像令牌多超过2倍。在标准新视角合成基准上评估显示,其性能显著优于当前最先进水平,PSNR提升超过2.4dB,LPIPS降低超过40%。进一步扩展至逆渲染任务,在广泛使用的基准上进行定性和定量评估,均显示纹理与几何细节一致性提升,达到或超越最先进密集视图优化方法的LPIPS表现。代码与模型权重可在项目页面获取。
原文摘要 · Abstract (English)
We introduce the Large Sparse Reconstruction Model to study how scaling transformer context windows affects feed-forward 3D reconstruction. Although recent object-centric feed-forward methods produce robust, high-quality reconstructions, they still lag behind dense-view optimization in recovering fine-grained texture and appearance. We show that expanding the context window -- by substantially increasing the number of active object and image tokens -- narrows this gap and enables high-fidelity 3D object reconstruction and inverse rendering. To scale effectively, we adapt native sparse attention for 3D reconstruction with three key contributions: (1) an efficient coarse-to-fine pipeline that focuses computation on informative regions by predicting sparse high-resolution residuals; (2) a 3D-aware spatial routing mechanism that establishes accurate 2D-3D correspondences using explicit geometric distances rather than standard attention scores; and (3) a custom block-aware sequence-parallel strategy with an All-gather-KV protocol to balance dynamic, sparse workloads across GPUs. As a result, LSRM handles 20x more object tokens and >2x more image tokens than prior state-of-the-art (SOTA) methods. Extensive evaluations on standard novel-view synthesis benchmarks show substantial gains over the current SOTA, yielding >2.4dB higher PSNR and >40% lower LPIPS. Furthermore, when extending LSRM to inverse rendering, qualitative and quantitative evaluations on widely used benchmarks demonstrate consistent improvements in texture and geometry details, achieving an LPIPS that matches or exceeds that of SOTA dense-view optimization methods. Code and model weights are available on our project page.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。