让视频分割模型在视角大变化时仍保持准确，只用普通摄像头输入。
3AM: 3egment Anything with Geometric Consistency in Videos
- 融合3D几何特征与视觉特征，提升跨视角一致性
- 在ScanNet++上达90.6% IoU，追踪召回率71.7%，超越顶尖方法
- 无需相机位姿或深度图，仅需普通视频输入
现有视频对象分割方法如SAM2依赖外观特征，在大视角变化下表现不佳。传统3D实例分割需相机位姿和深度图，成本高。本文提出3AM，通过训练阶段引入MUSt3R的3D感知特征，设计轻量级特征融合模块，结合多层级几何对应信息。采用视野感知采样策略，确保帧间空间一致区域用于学习3D对应关系。推理时仅需RGB输入，无需相机参数或预处理。在包含宽基线运动的ScanNet++和Replica数据集上，3AM显著优于SAM2及同类方法，在ScanNet++选子集上达到90.6% IoU与71.7%追踪召回率，较最先进方法分别提升15.9和30.4个百分点。
原文摘要 · Abstract (English)
Video object segmentation methods like SAM2 achieve strong performance through memory-based architectures but struggle under large viewpoint changes due to reliance on appearance features. Traditional 3D instance segmentation methods address viewpoint consistency but require camera poses, depth maps, and expensive preprocessing. We introduce 3AM, a training-time enhancement that integrates 3D-aware features from MUSt3R into SAM2. Our lightweight Feature Merger fuses multi-level MUSt3R features that encode implicit geometric correspondence. Combined with SAM2's appearance features, the model achieves geometry-consistent recognition grounded in both spatial position and visual similarity. We propose a field-of-view aware sampling strategy ensuring frames observe spatially consistent object regions for reliable 3D correspondence learning. Critically, our method requires only RGB input at inference, with no camera poses or preprocessing. On challenging datasets with wide-baseline motion (ScanNet++, Replica), 3AM substantially outperforms SAM2 and extensions, achieving 90.6% IoU and 71.7% Tracking Recall on ScanNet++'s Selected Subset, improving over state-of-the-art VOS methods by +15.9 and +30.4 points. Project page: https://jayisaking.github.io/3AM-Page/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。