让视频多目标分割实时运行，目标越多越快
SAM-MT: Real-Time Interactive Multi-Target Video Segmentation
- 用显式查询并行表示多个目标，共享全局上下文
- 10个目标仍保持36帧以上速度，延迟不随目标数上升
- 适合需要快速交互分割多物体的场景
现代视频对象分割需跟踪并分割用户指定的目标。现有方法在单目标场景表现优异，但扩展到多目标时通常对每个目标重复处理，导致帧率下降且延迟随目标数无界增长。基于Segment Anything 2（SAM2），我们提出SAM-MT，将模型转化为实时交互式多目标视频分割框架。SAM-MT使用显式查询表示不同目标，同时共享全局上下文；采用解耦掩码注意力避免跨目标干扰，结合稀疏记忆实现稳定时序演化，并设计专用策略处理遮挡与重叠。该方法成功将延迟与目标数量解耦，在10个目标下仍保持超过36 FPS的实时性能，同时维持SAM2的鲁棒性。
原文摘要 · Abstract (English)
Modern Video Object Segmentation (VOS) involves tracking and segmenting user-specified targets. While recent approaches have achieved remarkable performance in single-target scenarios, extending them to multi-target settings typically involves replicating the single-target processing for each individual object, resulting in reduced frame rates (FPS) with unbounded latency as target count increases. Built upon Segment Anything 2 (SAM2), we propose SAM-MT, which addresses this by transforming the model into an interactive framework for real-time Multi-Target video segmentation. SAM-MT uses explicit queries to represent different individual targets, in parallel with a shared representation for global context. It employs decoupled masked attention to keep individual identities distinct from cross-target interference, and sparse memory for stable temporal evolution, along with specialized strategies for occlusion handling and overlap prevention. SAM-MT successfully decouples latency from the number of targets, achieving real-time speed on par with single-target baselines (>36 FPS for 10 targets) while maintaining SAM2's robust video segmentation performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。