提出头级时序合并方法，加速3D场景重建模型推理
HTTM: Head-wise Temporal Token Merging for Faster VGGT
- 按注意力头粒度分组合并时序特征，保留多头表达差异
- 在大场景下实现最高7倍加速，性能损失可忽略
- 无需训练，适合部署于高分辨率3D重建系统
视觉几何定位变压器（VGGT）是首个在单次前向传播中联合推断所有关键3D属性（相机位姿、深度与稠密几何）的模型。然而，其全局注意力机制需对所有视角的令牌进行全连接注意力计算，导致在长序列输入的大场景重建中产生显著延迟瓶颈。本文提出无训练的头级时序合并（HTTM）方法，以加速VGGT推理。现有合并方法在各注意力头间均匀合并，造成输出特征冗余，削弱模型表达能力。HTTM通过在多头粒度上合并令牌，保持头拼接后特征的独特性，并利用头级别观察到的空间局部性与时间对应关系，实现更高合并率与更低合并开销。实验表明，基于GPU的推理中，HTTM相比原VGGT最高提速7倍，且性能下降可忽略。
原文摘要 · Abstract (English)
The Visual Geometry Grounded Transformer (VGGT) marks a significant leap forward in 3D scene reconstruction, as it is the first model that directly infers all key 3D attributes (camera poses, depths, and dense geometry) jointly in one pass. However, this joint inference mechanism requires global attention layers that perform all-to-all attention computation on tokens from all views. For reconstruction of large scenes with long-sequence inputs, this causes a significant latency bottleneck. In this paper, we propose head-wise temporal merging (HTTM), a training-free 3D token merging method for accelerating VGGT. Existing merging techniques merge tokens uniformly across different attention heads, resulting in identical tokens in the layers' output, which hinders the model's representational ability. HTTM tackles this problem by merging tokens in multi-head granularity, which preserves the uniqueness of feature tokens after head concatenation. Additionally, this enables HTTM to leverage the spatial locality and temporal correspondence observed at the head level to achieve higher merging ratios with lower merging costs compared to existing methods. Thus, HTTM achieves up to $7\times$ acceleration over the original VGGT with negligible performance drops in a GPU-based inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。