arXiv:2507.11893cs.CVcs.AI2025-07TPAMI被引 40

通过频率调制保留分割细节,解决下采样导致的高频信息丢失问题。

Spatial Frequency Modulation for Semantic Segmentation

论文配图:Spatial Frequency Modulation for Semantic Segmentation
图 1 · 摘自论文原文
  • 先调制高频特征至低频再下采样,上采样时再恢复。
  • 在Cityscapes上提升1.2%的mIoU,显著改善细节分割效果。
  • 适配卷积与Transformer架构,适用于多类视觉任务。

高空间频率信息(如纹理等细节)对语义分割精度至关重要。然而根据奈奎斯特-香农采样定理,通过步幅卷积等下采样层传播时,高频分量易受混叠或失真影响。为此,我们提出一种新型空间频率调制(SFM)方法:在下采样前将高频特征调制到较低频率,上采样时再解调恢复。具体地,采用自适应重采样(ARS)实现调制,并设计轻量级模块,在高频区域密集采样以放大信号,依据频率缩放特性降低其频率。同时提出多尺度自适应上采样(MSAU),通过非均匀上采样解调并恢复高频信息,显式利用多尺度下密集与稀疏重采样区域间的交互。两个模块可无缝集成于各类网络架构,从卷积神经网络扩展至Transformer。特征可视化与分析表明,该方法有效缓解了混叠现象,并在解调后成功保留细节。最后,我们将SFM拓展至图像分类、对抗鲁棒性、实例分割和全景分割任务,验证了其广泛适用性与有效性。代码已开源:https://github.com/Linwei-Chen/SFM。

原文摘要 · Abstract (English)

High spatial frequency information, including fine details like textures, significantly contributes to the accuracy of semantic segmentation. However, according to the Nyquist-Shannon Sampling Theorem, high-frequency components are vulnerable to aliasing or distortion when propagating through downsampling layers such as strided-convolution. Here, we propose a novel Spatial Frequency Modulation (SFM) that modulates high-frequency features to a lower frequency before downsampling and then demodulates them back during upsampling. Specifically, we implement modulation through adaptive resampling (ARS) and design a lightweight add-on that can densely sample the high-frequency areas to scale up the signal, thereby lowering its frequency in accordance with the Frequency Scaling Property. We also propose Multi-Scale Adaptive Upsampling (MSAU) to demodulate the modulated feature and recover high-frequency information through non-uniform upsampling This module further improves segmentation by explicitly exploiting information interaction between densely and sparsely resampled areas at multiple scales. Both modules can seamlessly integrate with various architectures, extending from convolutional neural networks to transformers. Feature visualization and analysis confirm that our method effectively alleviates aliasing while successfully retaining details after demodulation. Finally, we validate the broad applicability and effectiveness of SFM by extending it to image classification, adversarial robustness, instance segmentation, and panoptic segmentation tasks. The code is available at https://github.com/Linwei-Chen/SFM.

语义分割频率调制细节保留多尺度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。