无需参考图即可实现精准风格化图像生成
PokeFusion Attention: A Lightweight Cross-Attention Mechanism for Style-Conditioned Image Generation
- 将风格建模为可学习的分布先验,直接融入解码器跨注意力
- 在基准测试中提升风格保真度与结构一致性,参数开销极低
- 适合追求轻量化、免参考图的风格化图像生成场景
以扩散模型为基础的风格条件文本到图像生成,需兼顾角色结构稳定性和跨多样提示的一致性精细风格表达。现有方法或仅依赖文本提示(常不足以指定视觉风格),或引入需推理时依赖外部参考图的适配器,增加系统复杂度并限制部署灵活性。本文提出轻量级解码器级跨注意力机制PokeFusion Attention,将风格建模为可学习的分布先验而非实例级条件。该方法在扩散解码器内直接融合文本语义与学习到的风格嵌入,实现有效风格化生成,且推理时不需参考图像。仅训练跨注意力层和紧凑风格投影模块,预训练扩散主干保持冻结,具备参数高效与即插即用特性。在风格化角色生成基准上实验表明,相比代表性适配器基线,该方法在风格保真度、语义对齐与结构一致性上均有提升,同时维持低参数开销与简单推理流程。
原文摘要 · Abstract (English)
Style-conditioned text-to-image (T2I) generation with diffusion models requires both stable character structure and consistent, fine-grained style expression across diverse prompts. Existing approaches either rely on text-only prompting, which is often insufficient to specify visual style, or introduce reference-based adapters that depend on external images at inference time, increasing system complexity and limiting deployment flexibility. We propose PokeFusion Attention, a lightweight decoder-level cross-attention mechanism that models style as a learned distributional prior rather than instance-level conditioning. The method integrates textual semantics with learned style embeddings directly within the diffusion decoder, enabling effective stylized generation without requiring reference images at inference time. Only the cross-attention layers and a compact style projection module are trained, while the pretrained diffusion backbone remains frozen, resulting in a parameter-efficient and plug-and-play design. Experiments on a stylized character generation benchmark demonstrate that the proposed method improves style fidelity, semantic alignment, and structural consistency compared with representative adapter-based baselines, while maintaining low parameter overhead and simple inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。