针对360度视频分割中的畸变与记忆缺陷,提出轻量级改进方案。
PanoSAM2: Lightweight Distortion- and Memory-aware Adaptions of SAM2 for 360 Video Object Segmentation
- 设计全景感知解码器与畸变引导损失,修复拼接边界畸变
- 引入长短时记忆模块,提升长时间对象一致性,准确率提升5.6-6.7点
- 保留SAM2易用提示设计,适合虚拟现实与机器人视觉应用
360视频目标分割(360VOS)旨在实现全景覆盖下的时序一致掩码预测,广泛应用于虚拟现实、增强现实与具身智能。由于缺乏高质量标注数据集,360VOS建模极具挑战。近期的分段一切模型(SAMs),特别是具备记忆模块的SAM2,展现出强大且可提示的视频目标分割能力。然而,直接使用SAM2进行360VOS会产生不合理结果,主要因360视频存在投影畸变、左右侧语义不一致以及SAM2记忆中对象掩码信息稀疏等问题。为此,本文提出PanoSAM2,一种基于轻量级畸变与记忆感知适应策略的新型360VOS框架,在保留SAM2用户友好提示设计的同时实现可靠分割。具体而言,为解决投影畸变和语义不一致问题,提出全景感知解码器,具备缝合一致感受野并支持迭代畸变修正,以保持0/360度边界连续性;同时引入畸变引导掩码损失,按畸变程度加权像素,强化拉伸区域与边界。为缓解对象稀疏问题,设计长短时记忆模块,通过紧凑长时对象指针重激活并对齐短时记忆,提升时间连贯性。大量实验表明,相较于SAM2,PanoSAM2在360VOTS上提升+5.6,在PanoVOS上提升+6.7,验证了方法的有效性。
原文摘要 · Abstract (English)
360 video object segmentation (360VOS) aims to predict temporally-consistent masks in 360 videos, offering full-scene coverage, benefiting applications, such as VR/AR and embodied AI. Learning 360VOS model is nontrivial due to the lack of high-quality labeled dataset. Recently, Segment Anything Models (SAMs), especially SAM2 -- with its design of memory module -- shows strong, promptable VOS capability. However, directly using SAM2 for 360VOS yields implausible results as 360 videos suffer from the projection distortion, semantic inconsistency of left-right sides, and sparse object mask information in SAM2's memory. To this end, we propose PanoSAM2, a novel 360VOS framework based on our lightweight distortion- and memory-aware adaptation strategies of SAM2 to achieve reliable 360VOS while retaining SAM2's user-friendly prompting design. Concretely, to tackle the projection distortion and semantic inconsistency issues, we propose a Pano-Aware Decoder with seam-consistent receptive fields and iterative distortion refinement to maintain continuity across the 0/360 degree boundary. Meanwhile, a Distortion-Guided Mask Loss is introduced to weight pixels by distortion magnitude, stressing stretched regions and boundaries. To address the object sparsity issue, we propose a Long-Short Memory Module to maintain a compact long-term object pointer to re-instantiate and align short-term memories, thereby enhancing temporal coherence. Extensive experiments show that PanoSAM2 yields substantial gains over SAM2: +5.6 on 360VOTS and +6.7 on PanoVOS, showing the effectiveness of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。