解决视频目标分割中文本描述不一致与上下文忽略问题
Multi-Context Temporal Consistent Modeling for Referring Video Object Segmentation
- 引入多上下文时序一致性模块，统一查询表示
- 在MeViS数据集上达47.6 J&F，显著提升分割精度
- 适用于各类Transformer架构，通用性强
指代视频对象分割旨在根据给定文本描述分割视频中的对应目标。现有基于Transformer的时序建模方法存在查询不一致和上下文考虑不足的问题：查询不一致导致视频中间帧出现不稳定掩码；上下文忽略使模型无法准确捕捉文本与实例间关系，造成错误分割。为此，本文提出多上下文时序一致性模块（MTCM），包含对齐器（Aligner）和多上下文增强器（MCE）。对齐器去除查询噪声并实现查询对齐，保障一致性；MCE通过融合多上下文信息预测与文本相关的查询。将MTCM应用于四种不同模型，均取得性能提升，尤其在MeViS数据集上达到47.6 J&F。代码已开源。
原文摘要 · Abstract (English)
Referring video object segmentation aims to segment objects within a video corresponding to a given text description. Existing transformer-based temporal modeling approaches face challenges related to query inconsistency and the limited consideration of context. Query inconsistency produces unstable masks of different objects in the middle of the video. The limited consideration of context leads to the segmentation of incorrect objects by failing to adequately account for the relationship between the given text and instances. To address these issues, we propose the Multi-context Temporal Consistency Module (MTCM), which consists of an Aligner and a Multi-Context Enhancer (MCE). The Aligner removes noise from queries and aligns them to achieve query consistency. The MCE predicts text-relevant queries by considering multi-context. We applied MTCM to four different models, increasing performance across all of them, particularly achieving 47.6 J&F on the MeViS. Code is available at https://github.com/Choi58/MTCM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。