提升视频分割精度与长期跟踪稳定性，解决边界模糊和记忆冗余问题。
HQ-SMem: Video Segmentation and Tracking Using Memory Efficient Object Embedding With Selective Update and Self-Supervised Distillation Feedback
- 结合SAM-HQ与候选选择，精修粗分割掩码边界。
- 动态智能记忆机制，仅存储关键帧，降低内存开销。
- 自监督蒸馏反馈实现动态更新，缓解遮挡与形变导致的漂移。
视频对象分割（VOS）是监控、自动驾驶、机器人及生成式视频编辑等众多计算机视觉应用的基础。然而，现有模型在精确掩码划分、可变形物体、拓扑变化物体、跟踪漂移及长视频序列方面仍存在挑战。本文提出HQ-SMem，一种基于智能记忆的高质量视频分割与跟踪方法，通过三项创新显著提升基础模型性能：(i) 结合高精度掩码的SAM-HQ与基于外观的候选选择，优化粗分割掩码，改善物体边界；(ii) 设计动态智能记忆机制，仅保留关键帧并丢弃冗余帧，提升长视频处理效率与内存利用率；(iii) 动态更新外观模型以应对复杂拓扑变化，减少视频全程中的漂移。在多个公开数据集及主流基线追踪器上的大量实验表明，该方法在VOTS 2024与VOTSt 2024数据集上始终位列前二；在Long Video Dataset与LVOS上刷新基准，验证其在长时间、多对象复杂动态场景下的有效性。
原文摘要 · Abstract (English)
Video Object Segmentation (VOS) is foundational to numerous computer vision applications, including surveillance, autonomous driving, robotics and generative video editing. However, existing VOS models often struggle with precise mask delineation, deformable objects, topologically transforming objects, tracking drift and long video sequences. In this paper, we introduce HQ-SMem, for High Quality video segmentation and tracking using Smart Memory, a novel method that enhances the performance of VOS base models by addressing these limitations. Our approach incorporates three key innovations: (i) leveraging SAM with High-Quality masks (SAM-HQ) alongside appearance-based candidate-selection to refine coarse segmentation masks, resulting in improved object boundaries; (ii) implementing a dynamic smart memory mechanism that selectively stores relevant key frames while discarding redundant ones, thereby optimizing memory usage and processing efficiency for long-term videos; and (iii) dynamically updating the appearance model to effectively handle complex topological object variations and reduce drift throughout the video. These contributions mitigate several limitations of existing VOS models including, coarse segmentations that mix-in background pixels, fixed memory update schedules, brittleness to drift and occlusions, and prompt ambiguity issues associated with SAM. Extensive experiments conducted on multiple public datasets and state-of-the-art base trackers demonstrate that our method consistently ranks among the top two on VOTS and VOTSt 2024 datasets. Moreover, HQ-SMem sets new benchmarks on Long Video Dataset and LVOS, showcasing its effectiveness in challenging scenarios characterized by complex multi-object dynamics over extended temporal durations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。