让扩散模型生成多物体图像时风格更统一,不需训练
Local Prompt Adaptation for Style-Consistent Multi-Object Generation in Diffusion Models
- 将提示词拆成内容和风格,分时段注入注意力层
- 在T2I任务中提升0.41%提示对齐度,风格一致性更强
- 无需训练、易集成,适合需要精准控制的生成场景
扩散模型已成为文本到图像生成的强大基础,能从自然语言提示生成高质量图像。然而,当提示包含多个对象及全局或局部风格指令时,输出常出现风格漂移且空间关系不连贯,限制了可控、风格一致场景生成的可靠性。本文提出轻量级、免训练的局部提示适配(LPA)方法,将提示词分解为内容与风格标记,并在特定去噪步骤中选择性注入U-Net的注意力层。通过早期条件化对象标记、后期注入风格标记,LPA在不增加训练成本的前提下提升了布局控制与风格一致性。在多种解析设置与注入窗口的消融实验中,最佳配置(仅使用晚期注入,300-650步窗口)表现最优。在T2I基准测试中,相比原始SDXL提升0.41%的CLIP提示对齐度,相比SD1.5提升0.34%,且无多样性损失;在自建的50条提示风格丰富基准上,分别取得+0.09%提示对齐与+0.08%风格对齐的增益。该方法具备模型无关性、易集成性,仅需单次配置变更,是实现可控、风格一致多物体生成的实用方案。
原文摘要 · Abstract (English)
Diffusion models have become a powerful backbone for text-to-image generation, producing high-quality visuals from natural language prompts. However, when prompts involve multiple objects alongside global or local style instructions, the outputs often drift in style and lose spatial coherence, limiting their reliability for controlled, style-consistent scene generation. We present Local Prompt Adaptation (LPA), a lightweight, training-free method that splits the prompt into content and style tokens, then injects them selectively into the U-Net's attention layers at chosen timesteps. By conditioning object tokens early and style tokens later in the denoising process, LPA improves both layout control and stylistic uniformity without additional training cost. We conduct extensive ablations across parser settings and injection windows, finding that the best configuration -- lpa late only with a 300-650 step window -- delivers the strongest balance of prompt alignment and style consistency. On the T2I benchmark, LPA improves CLIP-prompt alignment over vanilla SDXL by +0.41% and over SD1.5 by +0.34%, with no diversity loss. On our custom 50-prompt style-rich benchmark, LPA achieves +0.09% CLIP-prompt and +0.08% CLIP-style gains over baseline. Our method is model-agnostic, easy to integrate, and requires only a single configuration change, making it a practical choice for controllable, style-consistent multi-object generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。