用混合记忆提升长序列3D重建一致性,支持实时推理。
Mem3R: Streaming 3D Reconstruction with Hybrid Memory via Test-Time Training

- 分拆相机追踪与几何映射,分别用快速权重和显式令牌记忆。
- 在500~1000帧序列上,绝对轨迹误差降低39%,参数量减少至644M。
- 兼容现有优化策略,适合机器人与增强现实的持续感知场景。
流式3D感知适用于机器人和增强现实领域,需高效且一致地处理长时间视觉流。近期递归模型通过维持固定大小的状态实现线性时间推理,但受限于压缩潜在记忆容量,常出现漂移积累和时间遗忘问题。我们提出Mem3R,一种具有混合记忆设计的流式3D重建模型,将相机追踪与几何映射解耦以提升长序列的时间一致性。相机追踪采用轻量级MLP实现的隐式快速权重记忆,通过测试时训练更新;几何映射则维护基于标记的固定大小显式状态。相比CUT3R,该设计不仅显著提升长序列性能,还将模型参数量从793M降至644M。Mem3R支持为CUT3R开发的改进即插即用状态更新策略。具体地,集成TTT3R后,在500至1000帧序列上,绝对轨迹误差最多降低39%。性能提升亦扩展至视频深度估计和3D重建等下游任务,同时保持恒定的GPU内存占用和相当的推理吞吐量。
原文摘要 · Abstract (English)
Streaming 3D perception is well suited to robotics and augmented reality, where long visual streams must be processed efficiently and consistently. Recent recurrent models offer a promising solution by maintaining fixed-size states and enabling linear-time inference, but they often suffer from drift accumulation and temporal forgetting over long sequences due to the limited capacity of compressed latent memories. We propose Mem3R, a streaming 3D reconstruction model with a hybrid memory design that decouples camera tracking from geometric mapping to improve temporal consistency over long sequences. For camera tracking, Mem3R employs an implicit fast-weight memory implemented as a lightweight Multi-Layer Perceptron updated via Test-Time Training. For geometric mapping, Mem3R maintains an explicit token-based fixed-size state. Compared with CUT3R, this design not only significantly improves long-sequence performance but also reduces the model size from 793M to 644M parameters. Mem3R supports existing improved plug-and-play state update strategies developed for CUT3R. Specifically, integrating it with TTT3R decreases Absolute Trajectory Error by up to 39% over the base implementation on 500 to 1000 frame sequences. The resulting improvements also extend to other downstream tasks, including video depth estimation and 3D reconstruction, while preserving constant GPU memory usage and comparable inference throughput. Project page: https://lck666666.github.io/Mem3R/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。