无需追踪和位姿,快速重建动态街道场景
StreetForward: Perceiving Dynamic Street with Feedforward Causal Attention
- 用时序掩码注意力捕捉图像序列中的运动信息
- 联合优化静态与动态内容,实现高保真新视角生成
- 在Waymo数据集上表现优于现有方法,可零样本泛化
前馈重建对自动驾驶应用至关重要,快速场景重建可高效利用大规模驾驶数据集进行闭环仿真等下游任务,避免耗时的逐场景优化。本文提出StreetForward,一种无位姿、无追踪的前馈式动态街道重建框架。基于视觉几何接地变压器(VGGT)的交替注意力机制,我们设计了一种简单有效的时序掩码注意力模块,从图像序列中捕捉动态运动信息,生成具有运动感知的潜在表示。静态内容与动态实例均采用3D高斯点云统一建模,并通过时空一致性跨帧渲染联合优化,使模型能够推断每像素速度并生成新位姿和时间下的高保真新视角。我们在Waymo Open Dataset上训练和评估该模型,结果表明其在新视角合成和深度估计方面显著优于现有方法。此外,在CARLA及其他数据集上的零样本推理验证了方法的泛化能力。更多可视化见项目主页:https://streetforward.github.io。
原文摘要 · Abstract (English)
Feedforward reconstruction is crucial for autonomous driving applications, where rapid scene reconstruction enables efficient utilization of large-scale driving datasets in closed-loop simulation and other downstream tasks, eliminating the need for time-consuming per-scene optimization. We present StreetForward, a pose-free and tracker-free feedforward framework for dynamic street reconstruction. Building upon the alternating attention mechanism from Visual Geometry Grounded Transformer (VGGT), we propose a simple yet effective temporal mask attention module that captures dynamic motion information from image sequences and produces motion-aware latent representations. Static content and dynamic instances are represented uniformly with 3D Gaussian Splatting, and are optimized jointly by cross-frame rendering with spatio-temporal consistency, allowing the model to infer per-pixel velocities and produce high-fidelity novel views at new poses and times. We train and evaluate our model on the Waymo Open Dataset, demonstrating superior performance on novel view synthesis and depth estimation compared to existing methods. Furthermore, zero-shot inference on CARLA and other datasets validates the generalization capability of our approach. More visualizations are available on our project page: https://streetforward.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。