用循环变压器块迭代优化多视角3D重建,参数少却效果更好。
Déjà View: Looping Transformers for Multi-View 3D Reconstruction

- 用一个循环块重复处理视图特征,实现多步精炼。
- 在5个基准上超越或媲美大模型,参数量仅为几分之一。
- 显式迭代比增加参数更有效,适合资源受限场景。
近期前馈式3D重建变换器已扩展至超过十亿参数,遵循计算机视觉中模型容量增长的趋势。然而,新证据表明连续变换器层常表现得像相似操作的重复应用,而多视角重建变换器在解码器深度上逐步优化预测。我们提出,模型深度部分通过额外参数换取了迭代效果,但效率低下;因此,我们显式地将迭代引入架构设计。所提出的模型DéjàView,通过一个循环的变压器块对每视图特征进行K次精炼。仅需一次训练,即可在推理时调节K作为计算控制开关。该模型在涵盖室内、室外、物体中心和驾驶场景的五个重建基准上,表现匹配或超越显著更大的前馈基线,同时使用极少量的参数和相当或更低的计算量。重要的是,在相同训练数据和计算条件下,相同的循环块结构优于具有独立每步参数的变体,表明显式迭代不仅是容量的高效替代,更是多视角3D重建的更强归纳偏置。
原文摘要 · Abstract (English)
Recent feed-forward 3D reconstruction transformers have scaled to over a billion parameters, following the broader trend of increasing model capacity in computer vision. Yet emerging evidence suggests that contiguous transformer layers often behave like repeated applications of similar operations, and multi-view reconstruction transformers refine their predictions progressively across decoder depth. We posit that model depth partially buys iteration, paid for inefficiently in unique parameters, and instead make that iteration explicit in architecture. Our model, DéjàView, applies a single looped transformer block recurrently to per-view features for K refinement steps. Trained once, it exposes K as an inference-time compute knob, matching or outperforming substantially larger feed-forward baselines across five reconstruction benchmarks spanning indoor, outdoor, object-centric, and driving scenes, while using a fraction of their parameters and comparable or lower compute. Importantly, the same looped block formulation outperforms an otherwise identical variant with independent per-step parameters under matched training data and compute, suggesting that explicit iteration is not merely a compute-efficient substitute for capacity but a stronger inductive bias for multi-view 3D reconstruction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。