用因果Transformer实现高效3D重建，支持实时流式处理。
STream3R: Scalable Sequential 3D Reconstruction with Causal Transformer
- 将点云预测转为仅解码器的Transformer任务，用因果注意力逐帧处理。
- 在静态与动态场景上均超越现有方法，尤其在长序列中表现更优。
- 兼容大模型训练框架，适合大规模预训练和下游3D任务应用。
我们提出STream3R，一种将点云预测重新构造成仅解码器Transformer问题的新方法。现有主流多视角重建方法依赖昂贵的全局优化或简单但难以扩展的内存机制。STream3R引入流式框架，利用受现代语言建模启发的因果注意力，高效处理图像序列。通过在大规模3D数据集上学习几何先验，该方法在多样化且具有挑战性的场景（包括动态场景）中表现出良好泛化能力。大量实验表明，其在静态与动态场景基准测试中持续优于先前工作。此外，STream3R天然兼容大语言模型式训练基础设施，支持高效的大规模预训练与微调，适用于多种下游3D任务。结果表明，因果Transformer模型在在线3D感知中具有巨大潜力，为流式环境下的实时3D理解铺平道路。更多信息见项目页：https://nirvanalan.github.io/projects/stream3r。
原文摘要 · Abstract (English)
We present STream3R, a novel approach to 3D reconstruction that reformulates pointmap prediction as a decoder-only Transformer problem. Existing state-of-the-art methods for multi-view reconstruction either depend on expensive global optimization or rely on simplistic memory mechanisms that scale poorly with sequence length. In contrast, STream3R introduces an streaming framework that processes image sequences efficiently using causal attention, inspired by advances in modern language modeling. By learning geometric priors from large-scale 3D datasets, STream3R generalizes well to diverse and challenging scenarios, including dynamic scenes where traditional methods often fail. Extensive experiments show that our method consistently outperforms prior work across both static and dynamic scene benchmarks. Moreover, STream3R is inherently compatible with LLM-style training infrastructure, enabling efficient large-scale pretraining and fine-tuning for various downstream 3D tasks. Our results underscore the potential of causal Transformer models for online 3D perception, paving the way for real-time 3D understanding in streaming environments. More details can be found in our project page: https://nirvanalan.github.io/projects/stream3r.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。