无需大量数据,高效实现多主体风格迁移。
ICAS: IP Adapter and ControlNet-based Attention Structure for Multi-Subject Style Transfer Optimization
- 仅微调内容注入分支,保留身份语义并提升风格可控性。
- 融合IP-Adapter与ControlNet,保持全局布局与局部风格一致。
- 循环多主体嵌入机制,低数据下仍可稳定迁移,适合实用场景。
生成多主体风格化图像仍面临挑战,主要源于风格属性(如色彩、纹理、氛围、结构)定义模糊,以及在多个主体间一致应用的困难。尽管基于扩散模型的文本到图像方法取得了显著进展,现有方法通常依赖计算成本高昂的反演过程或大规模风格化数据集,且难以维持多主体语义保真度,推理成本较高。为此,本文提出ICAS(IP-Adapter与ControlNet融合注意力结构),一种高效可控的多主体风格迁移新框架。不进行全模型微调,而是自适应地仅优化预训练扩散模型的内容注入分支,从而保留身份特异性语义,增强风格控制能力。通过结合IP-Adapter实现风格自适应注入与ControlNet实现结构条件约束,确保全局布局忠实还原与局部风格精准合成。此外,ICAS引入循环多主体内容嵌入机制,在数据有限条件下无需大规模风格化语料库即可实现有效风格迁移。大量实验表明,ICAS在结构保留、风格一致性与推理效率方面均表现优异,为真实场景中的多主体风格迁移建立了新范式。
原文摘要 · Abstract (English)
Generating multi-subject stylized images remains a significant challenge due to the ambiguity in defining style attributes (e.g., color, texture, atmosphere, and structure) and the difficulty in consistently applying them across multiple subjects. Although recent diffusion-based text-to-image models have achieved remarkable progress, existing methods typically rely on computationally expensive inversion procedures or large-scale stylized datasets. Moreover, these methods often struggle with maintaining multi-subject semantic fidelity and are limited by high inference costs. To address these limitations, we propose ICAS (IP-Adapter and ControlNet-based Attention Structure), a novel framework for efficient and controllable multi-subject style transfer. Instead of full-model tuning, ICAS adaptively fine-tunes only the content injection branch of a pre-trained diffusion model, thereby preserving identity-specific semantics while enhancing style controllability. By combining IP-Adapter for adaptive style injection with ControlNet for structural conditioning, our framework ensures faithful global layout preservation alongside accurate local style synthesis. Furthermore, ICAS introduces a cyclic multi-subject content embedding mechanism, which enables effective style transfer under limited-data settings without the need for extensive stylized corpora. Extensive experiments show that ICAS achieves superior performance in structure preservation, style consistency, and inference efficiency, establishing a new paradigm for multi-subject style transfer in real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。