arXiv:2602.21780cs.CV2026-02被引 6

压缩视觉模型的内存缓存,让长时间视频3D重建更高效

XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression

  • 通过剪枝与量化联合压缩键值缓存,保持高精度
  • 内存占用降低4.42倍,推理速度提升5.48倍
  • 无需调参,适合长时序3D重建应用

基于学习的3D视觉几何模型随着大规模Transformer的发展显著进步。其中,StreamVGGT利用帧级因果注意力实现鲁棒高效的流式3D重建,但因多图像和长视频输入带来大量视觉令牌,导致键值(KV)缓存无界增长,内存消耗与推理延迟随输入帧数累积而上升,限制了其在长时域应用中的可扩展性。为此,我们提出XStreamVGGT,一种无需调参的方法,通过剪枝与量化系统性压缩KV缓存,实现极低内存开销的流式推理。具体地,利用高效的令牌重要性识别机制,在不破坏高性能注意力核(如FlashAttention)兼容性的前提下,对多帧输入产生的冗余KVs进行剪枝,使其符合固定缓存预算;同时,结合KV张量固有的分布特性,采用维度自适应量化进一步减少内存开销,且保持数值精度。大量实验表明,XStreamVGGT在性能几乎无损的前提下,内存使用减少4.42倍,推理加速5.48倍,支持实际可扩展的流式3D应用。代码已开源:https://github.com/ywh187/XStreamVGGT/

原文摘要 · Abstract (English)

Learning-based 3D visual geometry models have significantly advanced with the advent of large-scale transformers. Among these, StreamVGGT leverages frame-wise causal attention to deliver robust and efficient streaming 3D reconstruction. However, it suffers from unbounded growth in the Key-Value (KV) cache due to the massive influx of vision tokens from multi-image and long-video inputs, leading to increased memory consumption and inference latency as input frames accumulate. This ultimately limits its scalability for long-horizon applications. To address this gap, we propose XStreamVGGT, a tuning-free approach that seamlessly integrates pruning and quantization to systematically compress the KV cache, enabling extremely memory-efficient streaming inference. Specifically, redundant KVs generated from multi-frame inputs are initially pruned to conform to a fixed KV memory budget using an efficient token-importance identification mechanism that maintains full compatibility with high-performance attention kernels (e.g., FlashAttention). Additionally, leveraging the inherent distribution patterns of KV tensors, we apply dimension-adaptive KV quantization within the pruning pipeline to further minimize memory overhead while preserving numerical accuracy. Extensive evaluations show that XStreamVGGT achieves mostly negligible performance degradation while substantially reducing memory usage by 4.42$\times$ and accelerating inference by 5.48$\times$, enabling practical and scalable streaming 3D applications. The code is available at https://github.com/ywh187/XStreamVGGT/.

3D重建内存优化流式推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。