用流式Transformer实现低延迟3D几何重建，支持实时交互。
Streaming 4D Visual Geometry Transformer
- 采用因果Transformer在线处理视频流，缓存历史信息做隐式记忆。
- 在多个基准上实现更快推理速度，同时保持高质量空间一致性。
- 适合需要实时3D重建的系统，如AR/VR与机器人导航。
从视频中感知和重建3D几何是计算机视觉中的基础但具有挑战性的任务。为支持交互式和低延迟应用，我们提出一种流式视觉几何Transformer，其设计思想类似自回归大语言模型。采用简单高效的因果Transformer架构，以在线方式处理输入序列。通过时间因果注意力机制，并缓存历史键值作为隐式记忆，实现高效长时序3D重建。该设计可增量融合历史信息，同时保持高空间一致性，支持低延迟3D重建。为提升训练效率，我们从密集双向视觉几何接地变压器（VGGT）中蒸馏知识到因果模型。推理时，支持迁移大型语言模型中的优化高效注意力算子（如FlashAttention）。在多个3D几何感知基准上的大量实验表明，本模型在在线场景中显著提升推理速度，同时保持竞争力性能，推动可扩展、交互式3D视觉系统的发展。代码已公开：https://github.com/wzzheng/StreamVGGT。
原文摘要 · Abstract (English)
Perceiving and reconstructing 3D geometry from videos is a fundamental yet challenging computer vision task. To facilitate interactive and low-latency applications, we propose a streaming visual geometry transformer that shares a similar philosophy with autoregressive large language models. We explore a simple and efficient design and employ a causal transformer architecture to process the input sequence in an online manner. We use temporal causal attention and cache the historical keys and values as implicit memory to enable efficient streaming long-term 3D reconstruction. This design can handle low-latency 3D reconstruction by incrementally integrating historical information while maintaining high-quality spatial consistency. For efficient training, we propose to distill knowledge from the dense bidirectional visual geometry grounded transformer (VGGT) to our causal model. For inference, our model supports the migration of optimized efficient attention operators (e.g., FlashAttention) from large language models. Extensive experiments on various 3D geometry perception benchmarks demonstrate that our model enhances inference speed in online scenarios while maintaining competitive performance, thereby facilitating scalable and interactive 3D vision systems. Code is available at: https://github.com/wzzheng/StreamVGGT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。