用跨注意力提升人物图像生成,兼顾外观与姿态,速度远超扩散模型。
Enhanced Multi-Scale Cross-Attention for Person Image Generation

- 设计双分支网络分别捕捉人物外观与姿态特征,通过跨注意力机制双向优化。
- 在两个公开数据集上优于主流GAN方法,性能接近扩散模型但训练推理更快。
- 提出多尺度跨注意力与增强注意力模块,有效减少噪声,提升长距离关联建模能力。
本文提出一种基于跨注意力的生成对抗网络XingGAN,用于挑战性的人物图像生成任务。该模型包含两个生成分支,分别捕获人物的外观和形状特征,并引入两种新颖的跨注意力块,实现外观与形状嵌入的相互更新与改进。为学习不同尺度和子区域间人物姿态的长程相关性,设计了多尺度跨注意力块。针对跨注意力中独立相关性计算导致的注意力权重模糊问题,提出增强注意力(EA)模块以优化权重分布。此外,引入密集连接的协同注意力模块,在不同阶段高效融合外观与形状特征。在两个公开数据集上的大量实验表明,所提方法优于现有基于GAN的方法,性能达到扩散模型水平,且在训练与推理速度上显著快于扩散模型。
原文摘要 · Abstract (English)
In this paper, we propose a novel cross-attention-based generative adversarial network (GAN) for the challenging person image generation task. Cross-attention is a novel and intuitive multi-modal fusion method in which an attention/correlation matrix is calculated between two feature maps of different modalities. Specifically, we propose the novel XingGAN (or CrossingGAN), which consists of two generation branches that capture the person's appearance and shape, respectively. Moreover, we propose two novel cross-attention blocks to effectively transfer and update the person's shape and appearance embeddings for mutual improvement. This has not been considered by any other existing GAN-based image generation work. To further learn the long-range correlations between different person poses at different scales and sub-regions, we propose two novel multi-scale cross-attention blocks. To tackle the issue of independent correlation computations within the cross-attention mechanism leading to noisy and ambiguous attention weights, which hinder performance improvements, we propose a module called enhanced attention (EA). Lastly, we introduce a novel densely connected co-attention module to fuse appearance and shape features at different stages effectively. Extensive experiments on two public datasets demonstrate that the proposed method outperforms current GAN-based methods and performs on par with diffusion-based methods. However, our method is significantly faster than diffusion-based methods in both training and inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。