提出不对称令牌压缩方法,让3D重建模型处理千帧视频快28倍。
Spark3R: Asymmetric Token Reduction Makes Fast Feed-Forward 3D Reconstruction

- 区分查询令牌与键值令牌角色,分别采用不同压缩策略。
- 在1000帧输入下实现最高28倍加速,重建质量仍领先。
- 无需重训练,可直接插入多个现成3D重建模型使用。
基于视觉变换器的前馈式3D重建模型能从少量输入图像直接估计场景几何与相机位姿,但扩展到含数百或数千帧的视频输入时,受限于全局注意力层的二次计算开销。现有令牌合并方法通过压缩注意力层中的令牌序列加速模型,但对查询令牌与键值令牌采用统一压缩,忽略了它们在3D重建中功能上的差异。本文发现:查询令牌编码视图特异性几何请求,对压缩敏感;键值令牌代表共享场景上下文,可承受激进压缩。基于此洞察,我们提出Spark3R——一种无需训练的加速框架,通过为查询与键值令牌分配不同压缩因子,对查询令牌进行组内合并,对键值令牌采用轻量级剪枝。此外,Spark3R在各层自适应调整键值压缩因子,进一步优化质量与效率权衡。作为即插即用框架,无需重训练,可集成至VGGT、$π^3$、Depth-Anything-3和VGGT-$Ω$等多个预训练前馈式3D重建模型,在1,000帧输入上实现最高28倍加速,同时保持竞争力的重建质量。
原文摘要 · Abstract (English)
Feed-forward 3D reconstruction models based on Vision Transformers can directly estimate scene geometry and camera poses from a small set of input images, but scaling them to video inputs with hundreds or thousands of frames remains challenging due to the quadratic cost of global attention layers. Recent token-merging methods accelerate these models by compressing the token sequence within the global attention layers, but they apply a uniform reduction to query tokens and key-value tokens, ignoring their functionally distinct roles in 3D reconstruction. In this work, we identify a key property of feed-forward 3D reconstruction models: query tokens encode view-specific geometric requests and are sensitive to compression, while key-value tokens represent shared scene context and tolerate aggressive compression. Guided by this insight, we propose Spark3R, a training-free acceleration framework that decouples the compression of query tokens and key-value tokens by assigning distinct reduction factors, with intra-group token merging applied to query tokens and lightweight token pruning to key-value tokens. Additionally, Spark3R adaptively adjusts the key-value reduction factor across layers, further improving the quality-efficiency trade-off. As a plug-and-play framework requiring no retraining, Spark3R integrates directly into multiple pretrained feed-forward 3D reconstruction models, including VGGT, $π^3$, Depth-Anything-3, and VGGT-$Ω$, and achieves up to $28\times$ speedup on 1,000-frame inputs while maintaining competitive reconstruction quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。