通过测试时训练提升3D重建模型长序列泛化能力
TTT3R: 3D Reconstruction as Test-Time Training
- 将3D重建视为在线学习问题，利用记忆与观测对齐置信度自适应更新
- 在数千张图像上实现2倍全局位姿估计性能提升，保持20帧/秒速度
- 无需训练即可部署，仅需6GB显存，适合实时长序列3D重建场景
现代循环神经网络因其线性时间复杂度成为3D重建的有力架构，但在超出训练上下文长度时性能显著下降，表现出有限的长度泛化能力。本文从测试时训练（Test-Time Training）视角重新审视3D重建基础模型，将其设计视为在线学习问题。基于此，我们利用记忆状态与新观测之间的对齐置信度，推导出记忆更新的闭式学习率，以平衡历史信息保留与新观测适应。该无训练干预方法称为TTT3R，显著提升长度泛化能力，在全局位姿估计上较基线实现2倍提升，同时以20 FPS的速度处理数千张图像，仅需6 GB GPU内存。代码已公开于https://rover-xingyu.github.io/TTT3R。
原文摘要 · Abstract (English)
Modern Recurrent Neural Networks have become a competitive architecture for 3D reconstruction due to their linear-time complexity. However, their performance degrades significantly when applied beyond the training context length, revealing limited length generalization. In this work, we revisit the 3D reconstruction foundation models from a Test-Time Training perspective, framing their designs as an online learning problem. Building on this perspective, we leverage the alignment confidence between the memory state and incoming observations to derive a closed-form learning rate for memory updates, to balance between retaining historical information and adapting to new observations. This training-free intervention, termed TTT3R, substantially improves length generalization, achieving a $2\times$ improvement in global pose estimation over baselines, while operating at 20 FPS with just 6 GB of GPU memory to process thousands of images. Code is available in https://rover-xingyu.github.io/TTT3R
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。