arXiv:2409.10385cs.CV2024-09被引 17

用状态空间模型实现高效风格迁移,速度更快、内存更省。

Mamba-ST: State Space Model for Efficient Style Transfer

  • 用Mamba模型模拟交叉注意力,直接融合内容与风格特征
  • 在艺术风格迁移任务中,FID和ArtFID指标优于主流Transformer与扩散模型
  • 首次纯SSM架构完成风格迁移,无需额外模块,适合实时应用

风格迁移的目标是给定一张内容图像和一个风格源,生成一幅保留内容但具有风格源艺术表现的新图像。现有先进方法多采用Transformer或扩散模型,但存在计算负担重的问题:Transformer依赖自注意力与交叉注意力层,内存开销大;扩散模型推理时间长。为此,本文提出Mamba-ST,一种基于新兴状态空间模型(SSM)的新型设计。通过改造Mamba线性方程,使其能模拟交叉注意力行为,将两个独立嵌入流融合为单一输出,显著降低内存占用与时间复杂度。我们修改了Mamba内部公式,以支持双流输入与融合。据我们所知,这是首个在不引入交叉注意力或定制归一化层的前提下,将SSM方程直接应用于视觉风格迁移的任务。大量实验表明,该方法在风格迁移质量与效率上均优于Transformer与扩散模型,艺术风格一致性(ArtFID)与图像质量(FID)均有提升。代码已开源:https://github.com/FilippoBotti/MambaST。

原文摘要 · Abstract (English)

The goal of style transfer is, given a content image and a style source, generating a new image preserving the content but with the artistic representation of the style source. Most of the state-of-the-art architectures use transformers or diffusion-based models to perform this task, despite the heavy computational burden that they require. In particular, transformers use self- and cross-attention layers which have large memory footprint, while diffusion models require high inference time. To overcome the above, this paper explores a novel design of Mamba, an emergent State-Space Model (SSM), called Mamba-ST, to perform style transfer. To do so, we adapt Mamba linear equation to simulate the behavior of cross-attention layers, which are able to combine two separate embeddings into a single output, but drastically reducing memory usage and time complexity. We modified the Mamba's inner equations so to accept inputs from, and combine, two separate data streams. To the best of our knowledge, this is the first attempt to adapt the equations of SSMs to a vision task like style transfer without requiring any other module like cross-attention or custom normalization layers. An extensive set of experiments demonstrates the superiority and efficiency of our method in performing style transfer compared to transformers and diffusion models. Results show improved quality in terms of both ArtFID and FID metrics. Code is available at https://github.com/FilippoBotti/MambaST.

风格迁移状态空间模型高效生成Mamba

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。