提出可无限流式处理3D几何理解的模型,解决长期稳定与扩展性难题。
InfiniteVGGT: Visual Geometry Grounded Transformer for Endless Streams
- 用自回归记忆缓存实现滚动式持续学习
- 在10,000帧长序列上优于现有方法,无灾难性退化
- 适合需要长期视觉理解的自动驾驶、机器人系统
实现持久、大规模3D视觉几何理解的愿景受限于可扩展性与长期稳定性的矛盾。虽离线模型VGGT具备出色几何能力,但其批量处理方式无法用于实时系统。流式架构虽为理想方案,却难以支持真正无限时长输入或在长序列中出现灾难性漂移。本文提出InfiniteVGGT,一种因果视觉几何变压器,通过有界但自适应、持续表达的键值缓存实现滚动记忆。基于此,设计无需训练、不依赖注意力机制的智能剪枝策略,随新帧到来自动淘汰过时信息,实现记忆滚动。该模型兼容FlashAttention,首次在无限时长流式输入下实现高性能与强稳定性平衡。为验证其长期性能,引入首个可严格评估连续3D几何估计的Long3D基准,支持约10,000帧的超长序列测试。代码已开源。
原文摘要 · Abstract (English)
The grand vision of enabling persistent, large-scale 3D visual geometry understanding is shackled by the irreconcilable demands of scalability and long-term stability. While offline models like VGGT achieve inspiring geometry capability, their batch-based nature renders them irrelevant for live systems. Streaming architectures, though the intended solution for live operation, have proven inadequate. Existing methods either fail to support truly infinite-horizon inputs or suffer from catastrophic drift over long sequences. We shatter this long-standing dilemma with InfiniteVGGT, a causal visual geometry transformer that operationalizes the concept of a rolling memory through a bounded yet adaptive and perpetually expressive KV cache. Capitalizing on this, we devise a training-free, attention-agnostic pruning strategy that intelligently discards obsolete information, effectively ``rolling'' the memory forward with each new frame. Fully compatible with FlashAttention, InfiniteVGGT finally alleviates the compromise, enabling infinite-horizon streaming while outperforming existing streaming methods in long-term stability. The ultimate test for such a system is its performance over a truly infinite horizon, a capability that has been impossible to rigorously validate due to the lack of extremely long-term, continuous benchmarks. To address this critical gap, we introduce the Long3D benchmark, which, for the first time, enables a rigorous evaluation of continuous 3D geometry estimation on sequences about 10,000 frames. This provides the definitive evaluation platform for future research in long-term 3D geometry understanding. Code is available at: https://github.com/AutoLab-SAI-SJTU/InfiniteVGGT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。