用3D高斯点云显式记忆物体位置，提升视频分割准确率
Explicit Memory through Online 3D Gaussian Splatting Improves Class-Agnostic Video Segmentation
- 用在线3D高斯点云存储视频中物体位置信息
- 相比无记忆或隐式记忆，分割结果更准确稳定
- 适合需要长期一致性视频分割的场景
记住过去帧中物体段的位置对提升无类别视频分割的准确性和一致性很有帮助。现有算法要么不使用物体级记忆（如FastSAM），要么使用基于循环神经网络特征的隐式记忆（如SAM2）。本文通过引入显式的3D记忆，增强这两类模型，并证明改进后模型预测更准确、更一致。为此，我们开发了在线3D高斯点云（3DGS）技术，用于存储视频全程生成的物体级分割结果。基于该3DGS表示，提出FastSAM-Splat和SAM2-Splat两种融合方法，利用显式3DGS记忆优化基础模型的预测。消融实验验证了设计与超参数设置的有效性。真实世界和模拟基准测试结果显示，使用显式3D记忆的模型在准确性和一致性上均优于无记忆或仅依赖隐式神经网络记忆的模型。
原文摘要 · Abstract (English)
Remembering where object segments were predicted in the past is useful for improving the accuracy and consistency of class-agnostic video segmentation algorithms. Existing video segmentation algorithms typically use either no object-level memory (e.g. FastSAM) or they use implicit memories in the form of recurrent neural network features (e.g. SAM2). In this paper, we augment both types of segmentation models using an explicit 3D memory and show that the resulting models have more accurate and consistent predictions. For this, we develop an online 3D Gaussian Splatting (3DGS) technique to store predicted object-level segments generated throughout the duration of a video. Based on this 3DGS representation, a set of fusion techniques are developed, named FastSAM-Splat and SAM2-Splat, that use the explicit 3DGS memory to improve their respective foundation models' predictions. Ablation experiments are used to validate the proposed techniques' design and hyperparameter settings. Results from both real-world and simulated benchmarking experiments show that models which use explicit 3D memories result in more accurate and consistent predictions than those which use no memory or only implicit neural network memories. Project Page: https://topipari.com/projects/FastSAM-Splat/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。