通过压缩键值缓存，实现低内存的实时3D视觉建模。
XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression
- 联合剪枝与量化压缩键值缓存，固定内存占用。
- 内存减少4.42倍，推理速度提升5.48倍，性能几乎无损失。
- 适合需要持续处理视频流的3D重建场景。
基于学习的3D视觉几何模型受益于大规模Transformer。其中，StreamVGGT利用帧级因果注意力实现强流式重建，但存在键值（KV）缓存无界增长问题，导致随输入帧积累，内存消耗和推理延迟持续上升。本文提出XStreamVGGT，一种无需调优的方法，通过联合剪枝与量化系统压缩KV缓存，实现极低内存的流式推理。具体地，通过高效识别多视角输入中冗余的键向量进行剪枝，实现固定内存预算；同时利用KV张量的独特分布特性，引入量化进一步降低内存开销。大量实验表明，XStreamVGGT在性能几乎无损的前提下，内存使用减少4.42倍，推理加速5.48倍，显著提升流式3D应用的可扩展性与实用性。代码已开源：https://github.com/ywh187/XStreamVGGT/。
原文摘要 · Abstract (English)
Learning-based 3D visual geometry models have benefited substantially from large-scale transformers. Among these, StreamVGGT leverages frame-wise causal attention for strong streaming reconstruction, but suffers from unbounded KV cache growth, leading to escalating memory consumption and inference latency as input frames accumulate. We propose XStreamVGGT, a tuning-free approach that systematically compresses the KV cache through joint pruning and quantization, enabling extremely memory-efficient streaming inference. Specifically, redundant KVs originating from multi-view inputs are pruned through efficient token importance identification, enabling a fixed memory budget. Leveraging the unique distribution of KV tensors, we incorporate KV quantization to further reduce memory consumption. Extensive evaluations show that XStreamVGGT achieves mostly negligible performance degradation while substantially reducing memory usage by 4.42$\times$ and accelerating inference by 5.48$\times$, enabling scalable and practical streaming 3D applications. The code is available at https://github.com/ywh187/XStreamVGGT/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。