用稀疏注意力加速多视角重建，速度提升3倍且精度不变
Block-Sparse Global Attention for Efficient Multi-View Geometry Transformers
- 发现全局注意力中只有少量图像块间交互重要，据此设计块稀疏替代方案
- 在多个多视图基准上实现推理速度提升3倍以上，性能基本保持不变
- 无需重新训练，可直接接入VGGT、π³等现有模型，适合大规模图像处理
高效准确的多视角重建是计算机视觉中的关键任务。近期基于Transformer的模型如VGGT、π³和MapAnything以简单架构取得了显著性能。然而，其可扩展性受限于全局注意力的二次复杂度，处理大量图像时带来严重运行时瓶颈。本文通过实证分析发现，这些模型的全局注意力矩阵中，概率质量集中在对应跨视图几何对应关系的少数图像块-图像块交互上。基于此观察，并受大语言模型中稀疏化思想启发，我们提出一种无需训练的块稀疏注意力替换方法，采用高度优化的内核实现。该方法使推理速度提升超过3倍，同时保持相近的任务性能。在涵盖多种多视图基准的评测中，本方法可无缝集成至VGGT、π³和MapAnything等现有全局注意力架构中，显著提升对大规模图像集合的可扩展性。
原文摘要 · Abstract (English)
Efficient and accurate feed-forward multi-view reconstruction has long been an important task in computer vision. Recent transformer-based models like VGGT, $π^3$ and MapAnything have demonstrated remarkable performance with relatively simple architectures. However, their scalability is fundamentally constrained by the quadratic complexity of global attention, which imposes a significant runtime bottleneck when processing large image sets. In this work, we empirically analyze the global attention matrix of these models and observe that the probability mass concentrates on a small subset of patch-patch interactions corresponding to cross-view geometric correspondences. Building on this insight and inspired by recent advances in large language models, we propose a training-free, block-sparse replacement for dense global attention, implemented with highly optimized kernels. Our method accelerates inference by more than $3\times$ while maintaining comparable task performance. Evaluations on a comprehensive suite of multi-view benchmarks demonstrate that our approach seamlessly integrates into existing global attention-based architectures such as VGGT, $π^3$ , and MapAnything, while substantially improving scalability to large image collections.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。