arXiv:2605.17478cs.CV2026-05

解决视频3D重建中长期记忆丢失问题,实现稳定长序列建模。

Mamba-VGGT: Persistent Long-Sequence Video Geometry Grounded Transformer via External Sliding Window Mamba Memory

论文配图:Mamba-VGGT: Persistent Long-Sequence Video Geometry Grounded Transformer via External Sliding Window Mamba Memory
图 1 · 摘自论文原文
  • 引入滑动窗口状态空间记忆模块,持久保存全局几何先验。
  • 在长序列上显著降低轨迹累积误差,空间一致性提升明显。
  • 适合需要长时间稳定建模的3D场景重建任务。

视觉几何接地变换器(VGGT)在高保真3D场景重建中达到新基准。然而,随着序列长度增加,这些模型因全局注意力的二次复杂度导致灾难性几何遗忘和累积漂移,需截断时间窗口。为此,我们提出Mamba-VGGT,一种支持持续长时推理的增强型VGGT框架。核心贡献是滑动窗口马尔可夫(SWM)记忆模块,通过外部记忆令牌跨时间窗口保持显式长期记忆。该模块利用选择性状态空间建模,提炼并传播全局几何先验,有效突破传统Transformer的记忆瓶颈。为在不破坏预训练VGGT优化的空间特征的前提下融合长期时序线索,我们提出零初始化空间记忆注入器。该注入器采用零卷积层,自适应地将持久记忆融入补丁令牌流,确保结构稳定性与特征对齐。大量实验表明,本方法在维持空间一致性和减少轨迹累积误差方面显著优于现有VGGT基线。工作提供了一种可扩展、线性复杂度的几何接地世界建模方案,适用于大规模3D环境。

原文摘要 · Abstract (English)

Visual Geometry Grounded Transformers (VGGT) have set new benchmarks in high-fidelity 3D scene reconstruction. However, as the sequence length increases, these models suffer from catastrophic geometric forgetting and accumulation drift, primarily due to the quadratic complexity of global attention which necessitates truncated temporal windows. To overcome the resulting geometric drift, we present Mamba-VGGT, an enhanced VGGT framework capable of persistent long-range reasoning. Our key contribution is a Sliding Window Mamba (SWM) memory module that maintains an explicit external memory token across temporal windows. This module leverages selective state-space modeling to distill and propagate global geometric priors, effectively bypassing the memory constraints of traditional transformers. To integrate these long-term temporal cues without disrupting the highly optimized spatial features of the pre-trained VGGT, we propose a Zero-Init Spatial Memory Injector. Utilizing zero-convolutional layers, this injector adaptively fuses persistent memory into the patch token stream, ensuring structural stability and seamless feature alignment. Extensive experiments demonstrate that our approach significantly outperforms existing VGGT-based methods in maintaining spatial consistency and reducing trajectory accumulation errors. Our work provides a scalable, linear-complexity solution for geometry-grounded world modeling in extensive 3D environments.

3D重建长序列建模状态空间模型几何约束

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。