持续更新的3D感知模型，能实时构建动态场景的稠密重建
Continuous 3D Perception Model with Persistent State
- 基于时序状态的递归模型，随新图像持续更新3D状态
- 在线生成每帧像素级3D点云，支持动态场景累积重建
- 适用于视频或照片集，可推断未观测视角的隐藏区域
我们提出一个统一框架，能够解决多种3D任务。该方法采用具有状态的循环模型，随每张新观测图像持续更新其状态表示。给定图像流，此演化状态可在线生成每帧的度量尺度点云（每像素3D点），并将其积累为一致的稠密场景重建，随着新图像到来不断更新。我们的模型CUT3R（用于3D重建的连续更新变换器）捕捉了真实场景的丰富先验：不仅能从图像观测中准确预测点云，还能通过探测虚拟未观测视角推断场景的不可见区域。该方法简单且高度灵活，自然适应不同长度的输入，包括视频流或无序照片集，含静态与动态内容。我们在多种3D/4D任务上评估，各项表现达到竞争性或领先水平。
原文摘要 · Abstract (English)
We present a unified framework capable of solving a broad range of 3D tasks. Our approach features a stateful recurrent model that continuously updates its state representation with each new observation. Given a stream of images, this evolving state can be used to generate metric-scale pointmaps (per-pixel 3D points) for each new input in an online fashion. These pointmaps reside within a common coordinate system, and can be accumulated into a coherent, dense scene reconstruction that updates as new images arrive. Our model, called CUT3R (Continuous Updating Transformer for 3D Reconstruction), captures rich priors of real-world scenes: not only can it predict accurate pointmaps from image observations, but it can also infer unseen regions of the scene by probing at virtual, unobserved views. Our method is simple yet highly flexible, naturally accepting varying lengths of images that may be either video streams or unordered photo collections, containing both static and dynamic content. We evaluate our method on various 3D/4D tasks and demonstrate competitive or state-of-the-art performance in each. Project Page: https://cut3r.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。