arXiv:2410.08781cs.CV2024-10ICRA被引 6

让AI在视频中持续精准分割物体,提升机器人感知能力。

VideoSAM: Open-World Video Segmentation

  • 用相似性匹配和记忆机制实现跨帧物体关联
  • 在UVO、BURST等数据集上显著提升分割一致性
  • 适合需要长期稳定视频理解的机器人应用

视频分割对推进机器人与自动驾驶至关重要,尤其在开放世界场景中,连续感知与跨帧物体关联尤为关键。尽管静态图像分割的SAM表现优异,但将其扩展至视频面临两大挑战:一、SAM嵌入难以跨帧关联物体;二、分割粒度不一致。为此,我们提出VideoSAM,一种端到端框架,通过融合聚合主干RADIO实现基于相似性的物体关联,并引入带记忆机制的循环确认对传播以保持跟踪稳定。同时,在SAM解码器中加入自回归对象标记机制,确保跨帧分割粒度一致。该方法在UVO、BURST基准及RoboTAP机器人视频上进行广泛评估,验证了其在真实场景中的有效性与鲁棒性。代码将公开。

原文摘要 · Abstract (English)

Video segmentation is essential for advancing robotics and autonomous driving, particularly in open-world settings where continuous perception and object association across video frames are critical. While the Segment Anything Model (SAM) has excelled in static image segmentation, extending its capabilities to video segmentation poses significant challenges. We tackle two major hurdles: a) SAM's embedding limitations in associating objects across frames, and b) granularity inconsistencies in object segmentation. To this end, we introduce VideoSAM, an end-to-end framework designed to address these challenges by improving object tracking and segmentation consistency in dynamic environments. VideoSAM integrates an agglomerated backbone, RADIO, enabling object association through similarity metrics and introduces Cycle-ack-Pairs Propagation with a memory mechanism for stable object tracking. Additionally, we incorporate an autoregressive object-token mechanism within the SAM decoder to maintain consistent granularity across frames. Our method is extensively evaluated on the UVO and BURST benchmarks, and robotic videos from RoboTAP, demonstrating its effectiveness and robustness in real-world scenarios. All codes will be available.

视频分割物体跟踪机器人感知

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。