无需文字提示,实现风格与内容自然融合的跨域图像合成
AIComposer: Any Style and Content Image Composition via Feature Integration
- 通过特征集成与局部交叉注意力控制扩散过程
- LPIPS降低30.5%,CSD提升18.1%,效果显著优于现有方法
- 适用于无文本提示的实用图像合成,适合创意设计场景
图像合成在大规模预训练文本到图像扩散模型推动下取得显著进展。尽管同域合成已较成熟,跨域合成仍缺乏探索。主要挑战来自扩散模型的随机性及输入图像间风格差异,导致合成失败和伪影。此外,对文本提示的强依赖限制了实际应用。本文提出首个无需文本提示的跨域图像合成方法,实现自然风格迁移与无缝融合。方法高效稳健,保留扩散先验,仅需少量反向重建与前向去噪步骤,不需训练扩散模型。采用简单MLP融合前景与背景的CLIP特征,并通过局部交叉注意力调控扩散过程,有效保持前景内容,实现稳定风格化且无需预训练风格网络。最后,我们构建了一个包含多样内容与风格的基准数据集,填补跨域图像合成评测数据缺失的空白。定性和定量评估均显示,本方法优于当前最先进技术,LPIPS下降30.5%,CSD提升18.1%。代码与数据集见https://github.com/sherlhw/AIComposer。
原文摘要 · Abstract (English)
Image composition has advanced significantly with large-scale pre-trained T2I diffusion models. Despite progress in same-domain composition, cross-domain composition remains under-explored. The main challenges are the stochastic nature of diffusion models and the style gap between input images, leading to failures and artifacts. Additionally, heavy reliance on text prompts limits practical applications. This paper presents the first cross-domain image composition method that does not require text prompts, allowing natural stylization and seamless compositions. Our method is efficient and robust, preserving the diffusion prior, as it involves minor steps for backward inversion and forward denoising without training the diffuser. Our method also uses a simple multilayer perceptron network to integrate CLIP features from foreground and background, manipulating diffusion with a local cross-attention strategy. It effectively preserves foreground content while enabling stable stylization without a pre-stylization network. Finally, we create a benchmark dataset with diverse contents and styles for fair evaluation, addressing the lack of testing datasets for cross-domain image composition. Our method outperforms state-of-the-art techniques in both qualitative and quantitative evaluations, significantly improving the LPIPS score by 30.5% and the CSD metric by 18.1%. We believe our method will advance future research and applications. Code and benchmark at https://github.com/sherlhw/AIComposer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。