arXiv:2411.06318cs.CV2024-11中稿 · WACV 2025被引 21

用状态空间模型实现高效图像修复,更好捕捉长距离依赖。

SEM-Net: Efficient Pixel Modelling for image inpainting with Spatially Enhanced SSM

  • 基于状态空间模型在像素级建模,线性计算复杂度提升效率。
  • 在两个数据集上超越现有方法,显著增强空间一致性与长程依赖捕捉能力。
  • 适合需要高精度图像修复与跨任务泛化的研究者使用。

图像修复旨在根据图像已知区域的信息修复部分损坏的图像。实现语义合理的结果尤为困难,因重建区域需与语义一致区域具有相似模式,这要求模型具备强大的长距离依赖(LRDs)建模能力。现有基于卷积神经网络的方法受制于感受野增长缓慢,而基于Transformer的方法则依赖块级交互,难以有效捕捉长程依赖。为此,我们提出SEM-Net,一种新型视觉状态空间模型(SSM)网络,在像素级建模受损图像的同时,通过状态空间捕捉长距离依赖,实现线性计算复杂度。为解决SSM固有的空间感知缺失问题,引入Snake Mamba Block(SMB)与空间增强前馈网络。这些创新使SEM-Net在两个不同数据集上优于当前最优方法,显著提升长程依赖建模与空间一致性表现。此外,其在运动去模糊任务上也达到领先性能,验证了模型的通用性。代码将开源。

原文摘要 · Abstract (English)

Image inpainting aims to repair a partially damaged image based on the information from known regions of the images. \revise{Achieving semantically plausible inpainting results is particularly challenging because it requires the reconstructed regions to exhibit similar patterns to the semanticly consistent regions}. This requires a model with a strong capacity to capture long-range dependencies. Existing models struggle in this regard due to the slow growth of receptive field for Convolutional Neural Networks (CNNs) based methods and patch-level interactions in Transformer-based methods, which are ineffective for capturing long-range dependencies. Motivated by this, we propose SEM-Net, a novel visual State Space model (SSM) vision network, modelling corrupted images at the pixel level while capturing long-range dependencies (LRDs) in state space, achieving a linear computational complexity. To address the inherent lack of spatial awareness in SSM, we introduce the Snake Mamba Block (SMB) and Spatially-Enhanced Feedforward Network. These innovations enable SEM-Net to outperform state-of-the-art inpainting methods on two distinct datasets, showing significant improvements in capturing LRDs and enhancement in spatial consistency. Additionally, SEM-Net achieves state-of-the-art performance on motion deblurring, demonstrating its generalizability. Our source code will be released in https://github.com/ChrisChen1023/SEM-Net.

图像修复状态空间模型长距离依赖

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。