提出在线3D场景重建与理解新框架,处理千帧以上序列仍保持高效。
S2GS: Streaming Semantic Gaussian Splatting for Online Scene Understanding and Reconstruction
- 采用因果增量更新机制,不重算历史帧,实现持续在线建模。
- 处理1000+帧时,运行时间和显存增长缓慢,而传统方法80帧即显存溢出。
- 支持几何与语义解耦建模,适合长期视频流的实时场景理解任务。
现有针对长图像序列的联合场景理解与重建方法多为离线前馈式,需对不断增长的历史观测重复进行全局计算,导致运行时间和GPU显存随序列长度快速上升,限制了可扩展性。本文提出流式语义高斯点云(S2GS),一种严格因果、增量式的3D高斯语义场框架:不利用未来帧,持续更新场景几何、外观和实例级语义,无需重新处理历史帧,实现可扩展的在线联合重建与理解。S2GS采用几何-语义解耦的双主干设计:几何分支进行因果建模以驱动增量高斯更新;语义分支基于2D基础视觉模型与查询驱动解码器预测分割掩码和身份嵌入,并通过查询级对比对齐和轻量级在线关联实例记忆加以稳定。实验表明,S2GS在联合重建与理解基准上达到或超过强大多帧离线基线性能,且显著提升长时序可扩展性:在相同设置下处理超过1000帧,运行时间与显存增长缓慢,而离线全局处理基线通常在约80帧时即因显存不足而中断。
原文摘要 · Abstract (English)
Existing offline feed-forward methods for joint scene understanding and reconstruction on long image streams often repeatedly perform global computation over an ever-growing set of past observations, causing runtime and GPU memory to increase rapidly with sequence length and limiting scalability. We propose Streaming Semantic Gaussian Splatting (S2GS), a strictly causal, incremental 3D Gaussian semantic field framework: it does not leverage future frames and continuously updates scene geometry, appearance, and instance-level semantics without reprocessing historical frames, enabling scalable online joint reconstruction and understanding. S2GS adopts a geometry-semantic decoupled dual-backbone design: the geometry branch performs causal modeling to drive incremental Gaussian updates, while the semantic branch leverages a 2D foundation vision model and a query-driven decoder to predict segmentation masks and identity embeddings, further stabilized by query-level contrastive alignment and lightweight online association with an instance memory. Experiments show that S2GS matches or outperforms strong offline baselines on joint reconstruction-and-understanding benchmarks, while significantly improving long-horizon scalability: it processes 1,000+ frames with much slower growth in runtime and GPU memory, whereas offline global-processing baselines typically run out of memory at around 80 frames under the same setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。