在视觉几何变压器中提前剪枝令牌,显著降低推理延迟。
PaceVGGT: Pre-Alternating-Attention Token Pruning for Visual Geometry Transformers

- 提前在注意力层前剪枝令牌,基于DINO特征评估重要性
- 扫描网50数据集上延迟降低5.1倍,保持重建质量
- 适合需要低延迟3D任务的部署场景
视觉几何变换器(VGGT)是多个3D任务的强大前馈模型,但其交替注意力(AA)堆叠的总令牌数呈二次增长,导致长视频处理成本高昂。现有令牌压缩方法作用于AA内部,未能压缩进入AA的补丁网格。本文提出PaceVGGT,一种在冻结VGGT的首个AA块之前进行预剪枝的框架。通过轻量级令牌评分器,从DINO特征中估计每个令牌的重要性。该评分器先在未剪枝主干中对齐注意力目标进行蒸馏,再在下游相机、深度和点图损失下优化。每帧保留预算固定主干可见序列长度,重要性自适应的合并/剪枝策略在固定合并预算下保留高显著性帧内容。特征引导恢复模块重建预测头所需的密集空间网格。在ScanNet-50和7-Scenes上,PaceVGGT维持重建质量与延迟的前沿表现。在ScanNet-50上,当N=300时延迟比未修改的VGGT降低5.1倍;当N=1000时比LiteVGGT降低1.47倍。结果表明,预AA剪枝是加速冻结型VGGT类几何变换器的有效路径。
原文摘要 · Abstract (English)
Visual Geometry Transformer (VGGT) is a strong feed-forward model for multiple 3D tasks, but its Alternating-Attention (AA) stack scales quadratically in the total token count, making long clips expensive. Existing token-reduction accelerators operate inside AA, leaving the patch grid that enters AA uncompressed. We introduce PaceVGGT, a pre-AA token pruning framework that prunes DINO patch tokens before the first AA block of a frozen VGGT. PaceVGGT trains a lightweight Token Scorer that estimates per-token importance from DINO features. The scorer is first distilled against an AA-internal attention target from the unpruned backbone, then refined under downstream camera, depth, and point-map losses. A per-frame keep budget fixes the backbone-visible sequence length, while an importance-adaptive merge/prune assignment preserves residual content from high-saliency frames under a fixed total merge budget. A Feature-guided Restoration module reconstructs the dense spatial grid required by the prediction heads. On ScanNet-50 and 7-Scenes, PaceVGGT remains on the reconstruction quality--latency frontier while reducing inference latency. On ScanNet-50, it reduces latency by \(5.1\times\) over unmodified VGGT at \(N=300\) and \(1.47\times\) over LiteVGGT at \(N=1000\). These results identify pre-AA pruning as a viable acceleration route for frozen VGGT-style geometry transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。