无需训练即可将离线3D重建模型转为实时流处理,突破内存瓶颈。
LASER: Layer-wise Scale Alignment for Training-Free Streaming 4D Reconstruction
- 按层逐级对齐深度尺度,解决视频流中深度模糊问题
- 在RTX A6000上实现14帧/秒、峰值显存6GB的高效运行
- 适用于千米级长视频实时重建,适合自动驾驶等场景
近期的前馈重建模型(如VGGT和$π^3$)虽能达到优异重建质量,但因内存复杂度呈二次增长,无法处理视频流,限制了实际应用。现有流式方法通过学习记忆机制或因果注意力缓解此问题,但需大量重训练,且未能充分利用先进离线模型的几何先验。本文提出LASER,一种无需训练的框架,通过跨连续时间窗口对齐预测,将离线模型转化为流式系统。我们发现,简单的相似变换(Sim(3))对齐因层深错位而失效:单目尺度模糊导致不同场景层在窗口间深度比例不一致。为此,提出分层尺度对齐策略,将深度预测分段为离散层,计算每层尺度因子,并在相邻窗口与时间戳间传播。大量实验表明,LASER在相机位姿估计与点云重建质量上达到当前最优,同时在RTX A6000上实现14 FPS、6 GB峰值显存,支持千米级视频流的实用部署。
原文摘要 · Abstract (English)
Recent feed-forward reconstruction models like VGGT and $π^3$ achieve impressive reconstruction quality but cannot process streaming videos due to quadratic memory complexity, limiting their practical deployment. While existing streaming methods address this through learned memory mechanisms or causal attention, they require extensive retraining and may not fully leverage the strong geometric priors of state-of-the-art offline models. We propose LASER, a training-free framework that converts an offline reconstruction model into a streaming system by aligning predictions across consecutive temporal windows. We observe that simple similarity transformation ($\mathrm{Sim}(3)$) alignment fails due to layer depth misalignment: monocular scale ambiguity causes relative depth scales of different scene layers to vary inconsistently between windows. To address this, we introduce layer-wise scale alignment, which segments depth predictions into discrete layers, computes per-layer scale factors, and propagates them across both adjacent windows and timestamps. Extensive experiments show that LASER achieves state-of-the-art performance on camera pose estimation and point map reconstruction %quality with offline models while operating at 14 FPS with 6 GB peak memory on a RTX A6000 GPU, enabling practical deployment for kilometer-scale streaming videos. Project website: https://neu-vi.github.io/LASER/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。