通过几何感知的缓存合并技术，让VGGT处理千图场景速度提升10倍
LiteVGGT: Boosting Vanilla VGGT via Geometry-aware Cached Token Merging
- 利用图像局部区域的几何相关性，动态合并冗余特征令牌
- 跨层复用合并索引，实现10倍加速与显著内存降低
- 适合需要高效处理大规模3D重建的科研与工业应用
3D视觉基础模型如视觉几何定位变压器（VGGT）在几何感知方面取得显著进展。然而，长序列处理耗时且内存占用高，限制了其在数百张以上图像的大规模场景中的应用。为此，我们提出LiteVGGT，实现最高10倍加速和显著内存减少，使1000张图像场景的高效处理成为可能。我们得出两个关键洞察：(1) 局部图像区域的令牌具有固有的几何关联性，导致高相似度和计算冗余；(2) 相邻网络层间的令牌相似性保持稳定，支持合并决策的可重用性。基于此，设计了一种名为几何感知缓存令牌合并的简单高效策略。通过分析每个令牌的几何重要性，优化锚点令牌选择以更好保留重建关键信息。同时，缓存并复用跨层合并索引，大幅降低延迟且精度损失极小。该策略保留VGGT核心性能，支持高效微调和FP8量化以进一步提升效率。大量实验验证了LiteVGGT的有效性、可扩展性和鲁棒性。
原文摘要 · Abstract (English)
3D vision foundation models like Visual Geometry Grounded Transformer (VGGT) have advanced greatly in geometric perception. However, it is time-consuming and memory-intensive for long sequences, limiting application to large-scale scenes beyond hundreds of images. To address this, we propose LiteVGGT, achieving up to 10x speedup and substantial memory reduction, enabling efficient processing of 1000-image scenes. We derive two key insights for 3D reconstruction: (1) tokens from local image regions have inherent geometric correlations, leading to high similarity and computational redundancy; (2) token similarity across adjacent network layers remains stable, allowing for reusable merge decisions. Guided by these, we design a simple yet efficient strategy, dubbed geometry-aware cached token merging. We analyze each token's geometric importance, optimizing anchor token selection to better preserve key information for reconstruction. We also cache and reuse merge indices across layers, substantially reducing latency with minimal accuracy impact. This strategy retains VGGT's core performance, enabling efficient fine-tuning and FP8 quantization for further gains. Extensive experiments validate LiteVGGT's effectiveness, scalability, and robustness. Project page: https://garlicba.github.io/LiteVGGT/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。