改进状态空间模型,让视频语义分割更精准。
RS-SSM: Refining Forgotten Specifics in State Space Model for Video Semantic Segmentation
- 设计通道注意力机制提取遗忘的细节特征
- 通过反向遗忘门自适应修复关键信息
- 适合需要高精度像素级分割的场景
近期状态空间模型通过线性复杂度的状态压缩实现了高效的视频分割。然而,视频语义分割(VSS)需具备像素级时空建模能力以保持语义对象的时间一致性。尽管状态空间模型能保留通用语义信息,但固定大小的状态空间不可避免地遗忘特定信息,限制了其在像素级分割上的表现。为此,本文提出一种修正遗忘细节的状态空间模型(RS-SSM),通过互补方式重构被遗忘的时空细节。具体地,设计通道幅度感知器(CwAP)以提取并对齐状态空间中特定信息的分布特征;同时提出遗忘门信息重构器(FGIR),基于特定信息分布自适应地反转并优化状态空间模型中的遗忘门矩阵。由此,RS-SSM利用反向遗忘门互补修复压缩过程丢失的细节信息,显著提升模型在时空像素级分割上的能力。在四个VSS基准测试上,本方法实现领先性能的同时保持高效计算效率。
原文摘要 · Abstract (English)
Recently, state space models have demonstrated efficient video segmentation through linear-complexity state space compression. However, Video Semantic Segmentation (VSS) requires pixel-level spatiotemporal modeling capabilities to maintain temporal consistency in segmentation of semantic objects. While state space models can preserve common semantic information during state space compression, the fixed-size state space inevitably forgets specific information, which limits the models' capability for pixel-level segmentation. To tackle the above issue, we proposed a Refining Specifics State Space Model approach (RS-SSM) for video semantic segmentation, which performs complementary refining of forgotten spatiotemporal specifics. Specifically, a Channel-wise Amplitude Perceptron (CwAP) is designed to extract and align the distribution characteristics of specific information in the state space. Besides, a Forgetting Gate Information Refiner (FGIR) is proposed to adaptively invert and refine the forgetting gate matrix in the state space model based on the specific information distribution. Consequently, our RS-SSM leverages the inverted forgetting gate to complementarily refine the specific information forgotten during state space compression, thereby enhancing the model's capability for spatiotemporal pixel-level segmentation. Extensive experiments on four VSS benchmarks demonstrate that our RS-SSM achieves state-of-the-art performance while maintaining high computational efficiency. The code is available at https://github.com/zhoujiahuan1991/CVPR2026-RS-SSM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。