改进视频目标分割的内存机制,提升复杂场景下的稳定性。
Addressing Issues with Working Memory in Video Object Segmentation
- 引入决策函数判断是否更新工作内存,避免错误信息干扰。
- 在存在镜头突变等异常情况下,性能显著优于原模型。
- 适用于需要鲁棒性分割的现实视频应用,如监控与自动驾驶。
当前先进的视频目标分割(VOS)模型通过关联当前未标注图像与历史图像-掩码关系来预测目标掩码,其内部记忆状态称为工作内存缓冲区。尽管这些模型在干净视频上表现优异,但对前序帧的依赖可能导致错误累积。基于亲和力的算法隐含了连续帧间存在时间连续性的归纳偏置。为应对目标视角不一致的情况(如突然镜头切换、帧插入、极端上下文变化),需对工作内存更新机制进行算法调整,避免无关帧写入内存。本文提出一种简单算法,可无缝集成至任意基于工作内存的VOS模型,有效提升在包含帧插入等异常情况下的性能。核心贡献是一个决策函数,根据检测到的剧烈变化判断对象是否已离屏,从而决定是否更新记忆。该方法显著增强了现有VOS模型在真实场景中的实用性。
原文摘要 · Abstract (English)
Contemporary state-of-the-art video object segmentation (VOS) models compare incoming unannotated images to a history of image-mask relations via affinity or cross-attention to predict object masks. We refer to the internal memory state of the initial image-mask pair and past image-masks as a working memory buffer. While the current state of the art models perform very well on clean video data, their reliance on a working memory of previous frames leaves room for error. Affinity-based algorithms include the inductive bias that there is temporal continuity between consecutive frames. To account for inconsistent camera views of the desired object, working memory models need an algorithmic modification that regulates the memory updates and avoid writing irrelevant frames into working memory. A simple algorithmic change is proposed that can be applied to any existing working memory-based VOS model to improve performance on inconsistent views, such as sudden camera cuts, frame interjections, and extreme context changes. The resulting model performances show significant improvement on video data with these frame interjections over the same model without the algorithmic addition. Our contribution is a simple decision function that determines whether working memory should be updated based on the detection of sudden, extreme changes and the assumption that the object is no longer in frame. By implementing algorithmic changes, such as this, we can increase the real-world applicability of current VOS models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。