提出双路框架,实现实时开集视频实例分割。
Segmenting, Fast and Slow: Real-Time Open-Vocabulary Video Instance Segmentation with Dual-Path Processing

- 分快慢两路:关键帧用对象级模型,后续帧用轻量快速网络
- 快分支延迟比MOBIUS低14倍,精度仍达主流水平
- 适合移动端实时视频分割,尤其对算力有限场景友好
受DETR启发的以对象为中心的模型已成为开集视频实例分割(OV-VIS)的主流范式。尽管近期工作已降低像素解码、文本模态融合和对象解码的计算开销,使这类架构更适合移动设备,但在高帧率下实现真正的实时端侧推理仍是未解难题。本文提出SegFS,一种双流快速-慢速框架,显著提升效率而不损失精度。在稀疏关键帧上,使用开集对象级模型预测实例级表征;这些表征被投影回主干特征空间,用于条件化一个轻量快速网络,以高效重定位并分割后续帧中的实例。通过将实例传播从对象解码转移到特征空间条件化,该方法解耦了多模态语义理解与密集掩码预测,实现了高效的时间传播。所提快速分支相较面向移动端的MOBIUS模型,延迟最高降低14倍,同时在标准OV-VIS基准上保持竞争力的分割性能。
原文摘要 · Abstract (English)
Object-centric models inspired by DETR have become the dominant paradigm for open-vocabulary video instance segmentation (OV-VIS). While recent efforts have reduced the computational cost of pixel decoding, textual modality fusion, and object decoding to make these architectures more suitable for mobile devices, real-time on-device inference at high frame rates remains an open challenge. In this paper, we introduce SegFS, a dual-stream fast-slow framework that significantly improves efficiency without sacrificing accuracy. On sparse keyframes, an open-vocabulary object-based model predicts instance-level representations. These representations are then projected back into the backbone feature space to condition a lightweight fast network, which efficiently relocalizes and segments the instances in subsequent frames. By shifting instance propagation from object decoding to feature-space conditioning, our approach decouples multimodal semantic understanding from dense mask prediction and enables efficient temporal propagation. The proposed fast branch achieves up to 14x lower latency than the mobile-oriented MOBIUS model, while maintaining competitive segmentation performance on standard OV-VIS benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。