无需训练的视频分割方法，通过动态记忆与时空解耦提升稳定性
Training-Free Spatio-temporal Decoupled Reasoning Video Segmentation with Adaptive Object Memory
- 不微调大模型，仅用预训练模型实现推理
- 在5个数据集上达到领先性能，尤其在长视频中表现稳定
- 适合需要低资源部署的实时视频分割场景
推理视频对象分割（ReasonVOS）是一项挑战性任务，要求在复杂文本输入下对视频序列中的对象进行稳定分割。现有方法需微调多模态大语言模型（MLLM），消耗大量资源；且部分方法在时空处理上耦合紧密，影响时序稳定性。为此，我们提出无需训练的时空解耦推理视频分割框架SDAM，其包含自适应对象记忆模块，根据运动线索选择并记忆关键对象；同时引入时空解耦机制，在空间域实现目标对象精准定位，在时间域利用关键对象信息驱动跨帧稳定传播。该方法在五大数据集（Ref-YouTubeVOS、Ref-DAVIS17、MeViS、ReasonVOS、ReVOS）上取得优异效果，优于依赖微调的现有方法。
原文摘要 · Abstract (English)
Reasoning Video Object Segmentation (ReasonVOS) is a challenging task that requires stable object segmentation across video sequences using implicit and complex textual inputs. Previous methods fine-tune Multimodal Large Language Models (MLLMs) to produce segmentation outputs, which demand substantial resources. Additionally, some existing methods are coupled in the processing of spatio-temporal information, which affects the temporal stability of the model to some extent. To address these issues, we propose Training-Free \textbf{S}patio-temporal \textbf{D}ecoupled Reasoning Video Segmentation with \textbf{A}daptive Object \textbf{M}emory (SDAM). We aim to design a training-free reasoning video segmentation framework that outperforms existing methods requiring fine-tuning, using only pre-trained models. Meanwhile, we propose an Adaptive Object Memory module that selects and memorizes key objects based on motion cues in different video sequences. Finally, we propose Spatio-temporal Decoupling for stable temporal propagation. In the spatial domain, we achieve precise localization and segmentation of target objects, while in the temporal domain, we leverage key object temporal information to drive stable cross-frame propagation. Our method achieves excellent results on five benchmark datasets, including Ref-YouTubeVOS, Ref-DAVIS17, MeViS, ReasonVOS, and ReVOS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。