arXiv:2603.14232cs.CV2026-03

提出在线3D场景重建与理解新框架,处理千帧以上序列仍保持高效。

S2GS: Streaming Semantic Gaussian Splatting for Online Scene Understanding and Reconstruction

  • 采用因果增量更新机制,不重算历史帧,实现持续在线建模。
  • 处理1000+帧时,运行时间和显存增长缓慢,而传统方法80帧即显存溢出。
  • 支持几何与语义解耦建模,适合长期视频流的实时场景理解任务。

现有针对长图像序列的联合场景理解与重建方法多为离线前馈式,需对不断增长的历史观测重复进行全局计算,导致运行时间和GPU显存随序列长度快速上升,限制了可扩展性。本文提出流式语义高斯点云(S2GS),一种严格因果、增量式的3D高斯语义场框架:不利用未来帧,持续更新场景几何、外观和实例级语义,无需重新处理历史帧,实现可扩展的在线联合重建与理解。S2GS采用几何-语义解耦的双主干设计:几何分支进行因果建模以驱动增量高斯更新;语义分支基于2D基础视觉模型与查询驱动解码器预测分割掩码和身份嵌入,并通过查询级对比对齐和轻量级在线关联实例记忆加以稳定。实验表明,S2GS在联合重建与理解基准上达到或超过强大多帧离线基线性能,且显著提升长时序可扩展性:在相同设置下处理超过1000帧,运行时间与显存增长缓慢,而离线全局处理基线通常在约80帧时即因显存不足而中断。

原文摘要 · Abstract (English)

Existing offline feed-forward methods for joint scene understanding and reconstruction on long image streams often repeatedly perform global computation over an ever-growing set of past observations, causing runtime and GPU memory to increase rapidly with sequence length and limiting scalability. We propose Streaming Semantic Gaussian Splatting (S2GS), a strictly causal, incremental 3D Gaussian semantic field framework: it does not leverage future frames and continuously updates scene geometry, appearance, and instance-level semantics without reprocessing historical frames, enabling scalable online joint reconstruction and understanding. S2GS adopts a geometry-semantic decoupled dual-backbone design: the geometry branch performs causal modeling to drive incremental Gaussian updates, while the semantic branch leverages a 2D foundation vision model and a query-driven decoder to predict segmentation masks and identity embeddings, further stabilized by query-level contrastive alignment and lightweight online association with an instance memory. Experiments show that S2GS matches or outperforms strong offline baselines on joint reconstruction-and-understanding benchmarks, while significantly improving long-horizon scalability: it processes 1,000+ frames with much slower growth in runtime and GPU memory, whereas offline global-processing baselines typically run out of memory at around 80 frames under the same setting.

3D重建在线学习语义建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。