解析多视图Transformer的3D表示机制，揭示其内部工作原理
Understanding Multi-View Transformers
- 通过残差连接探查模型层间隐状态演化
- 发现模型通过几何重构逐步优化对应点匹配
- 适合关注3D视觉模型可解释性与安全应用的研究者
多视图Transformer（如DUSt3R）正革新3D视觉任务，以前馈方式解决三维问题。然而，与以往基于优化的流程不同，这类模型的内部机制尚不清晰，其黑箱特性使超越数据规模的改进变得困难，并限制了其在安全与可靠性关键场景中的应用。本文提出一种方法，通过分析多视图Transformer各层残差连接中的3D表示，研究了DUSt3R的一个变体。结果揭示了潜在状态在各网络块间的演化过程，明确了各层的作用，并指出其与具有更强显式全局位姿归纳偏置方法的区别。此外，我们发现该变体模型所估计的对应关系会通过重建的几何信息进行精炼。分析代码已公开于 https://github.com/JulienGaubil/und3rstand。
原文摘要 · Abstract (English)
Multi-view transformers such as DUSt3R are revolutionizing 3D vision by solving 3D tasks in a feed-forward manner. However, contrary to previous optimization-based pipelines, the inner mechanisms of multi-view transformers are unclear. Their black-box nature makes further improvements beyond data scaling challenging and complicates usage in safety- and reliability-critical applications. Here, we present an approach for probing and visualizing 3D representations from the residual connections of the multi-view transformers' layers. In this manner, we investigate a variant of the DUSt3R model, shedding light on the development of its latent state across blocks, the role of the individual layers, and suggest how it differs from methods with stronger inductive biases of explicit global pose. Finally, we show that the investigated variant of DUSt3R estimates correspondences that are refined with reconstructed geometry. The code used for the analysis is available at https://github.com/JulienGaubil/und3rstand .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。