arXiv:2411.17061cs.CV2024-11被引 2

提出新型解码器SCASeg,用条带注意力提升分割效率与精度。

SCASeg: Strip Cross-Attention for Efficient Semantic Segmentation

  • 用编码器特征作查询,跨层融合多尺度特征图。
  • 在多个数据集上超越主流分割模型,计算量更低。
  • 适合需要高效高精度分割的实时应用场景。

视觉变换器(ViT)在计算机视觉中取得显著成功,其变体已广泛验证于各类下游任务,包括语义分割。然而,作为通用视觉编码器,ViT主干网络通常无法充分满足任务解码器的具体需求,为设计针对高效语义分割优化的解码器提供了契机。本文提出一种名为条带交叉注意力(SCASeg)的创新解码头,专为语义分割设计。不同于传统跳跃连接,SCASeg采用编码器与解码器阶段间的侧向连接,将编码器特征作为交叉注意力模块中的查询(Query)。此外,引入跨层块(CLB),整合来自不同编码器与解码器阶段的分层特征图,形成统一的键(Key)与值(Value)表示。CLB还融合了卷积的局部感知优势,使SCASeg能够跨多层捕捉全局与局部上下文依赖,增强多尺度特征交互,提升整体效率。为进一步优化计算效率,SCASeg将查询和键的通道压缩至一维,形成条带状结构,相比传统交叉注意力显著降低内存占用并提升推理速度。实验表明,SCASeg的可适配解码器在多种设置下表现优异,在ADE20K、Cityscapes、COCO-Stuff 164k和Pascal VOC2012等基准数据集上,即使在不同计算约束下,也优于主流分割架构。

原文摘要 · Abstract (English)

The Vision Transformer (ViT) has achieved notable success in computer vision, with its variants widely validated across various downstream tasks, including semantic segmentation. However, as general-purpose visual encoders, ViT backbones often do not fully address the specific requirements of task decoders, highlighting opportunities for designing decoders optimized for efficient semantic segmentation. This paper proposes Strip Cross-Attention (SCASeg), an innovative decoder head specifically designed for semantic segmentation. Instead of relying on the conventional skip connections, we utilize lateral connections between encoder and decoder stages, leveraging encoder features as Queries in cross-attention modules. Additionally, we introduce a Cross-Layer Block (CLB) that integrates hierarchical feature maps from various encoder and decoder stages to form a unified representation for Keys and Values. The CLB also incorporates the local perceptual strengths of convolution, enabling SCASeg to capture both global and local context dependencies across multiple layers, thus enhancing feature interaction at different scales and improving overall efficiency. To further optimize computational efficiency, SCASeg compresses the channels of queries and keys into one dimension, creating strip-like patterns that reduce memory usage and increase inference speed compared to traditional vanilla cross-attention. Experiments show that SCASeg's adaptable decoder delivers competitive performance across various setups, outperforming leading segmentation architectures on benchmark datasets, including ADE20K, Cityscapes, COCO-Stuff 164k, and Pascal VOC2012, even under diverse computational constraints.

语义分割Transformer高效模型注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。