融合卷积与注意力机制,提升语义分割的实时效率与精度。
ContextFormer: Redefining Efficiency in Semantic Segmentation
- 在瓶颈层结合CNN与ViT优势,动态建模多尺度特征。
- 在多个数据集上达到最新最高mIoU,实现实时推理性能突破。
- 适合追求高精度与低延迟的视觉应用开发者参考。
语义分割为图像像素分配标签,是计算机视觉中的关键任务但极具挑战。传统卷积方法虽擅长捕捉局部依赖,却难以处理长程关系;视觉变换器(ViTs)虽能有效建模全局上下文,但计算开销大,尤其对高分辨率输入。现有研究多聚焦编码器优化,而忽略了瓶颈层这一关键瓶颈——正是提升性能与效率的核心区域。本文提出ContextFormer,一种混合框架,通过在瓶颈层融合CNN与ViT优势,实现效率、准确率与鲁棒性的平衡,适用于实时语义分割。其高效性由三个协同模块驱动:分层多尺度表征的令牌金字塔提取模块(TPEM),支持动态尺度感知特征建模的Transformer与分支深度卷积块(Trans-BDC),以及增强空间与上下文一致性的特征融合模块(FMM)。在ADE20K、Pascal Context、CityScapes和COCO-Stuff数据集上的大量实验表明,ContextFormer显著优于现有模型,达成最新的mIoU成绩,树立了效率与性能的新基准。代码将在论文接受后公开。
原文摘要 · Abstract (English)
Semantic segmentation assigns labels to pixels in images, a critical yet challenging task in computer vision. Convolutional methods, although capturing local dependencies well, struggle with long-range relationships. Vision Transformers (ViTs) excel in global context capture but are hindered by high computational demands, especially for high-resolution inputs. Most research optimizes the encoder architecture, leaving the bottleneck underexplored - a key area for enhancing performance and efficiency. We propose ContextFormer, a hybrid framework leveraging the strengths of CNNs and ViTs in the bottleneck to balance efficiency, accuracy, and robustness for real-time semantic segmentation. The framework's efficiency is driven by three synergistic modules: the Token Pyramid Extraction Module (TPEM) for hierarchical multi-scale representation, the Transformer and Branched DepthwiseConv (Trans-BDC) block for dynamic scale-aware feature modeling, and the Feature Merging Module (FMM) for robust integration with enhanced spatial and contextual consistency. Extensive experiments on ADE20K, Pascal Context, CityScapes, and COCO-Stuff datasets show ContextFormer significantly outperforms existing models, achieving state-of-the-art mIoU scores, setting a new benchmark for efficiency and performance. The codes will be made publicly available upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。