让文生图模型读懂长段落描述,精准生成复杂场景。
Long-Text-to-Image Generation via Compositional Prompt Decomposition

- 将长文本拆解为多个组件,分步生成再融合
- 在500词以上提示下比基线高7.4%表现
- 无需微调,适配多种主流文生图模型
现代文生图(T2I)模型在处理复杂提示时表现优异,但面对描述性段落时难以捕捉关键细节。这一局限源于训练数据多为简洁标题。现有方法或通过微调长提示泛化性差,或压缩输入牺牲精度。本文提出PRISM,一种基于成分分解的框架,使预训练T2I模型能处理长序列输入。PRISM使用轻量模块提取长提示中的组成部分表示,T2I模型对每个组件独立预测噪声,并通过基于能量的联合机制合并输出,完成一次去噪步骤。在多种模型架构上评估显示,其性能可媲美在相同数据上微调的模型。此外,PRISM在挑战性公开基准上,对超过500词的提示表现优于基线7.4%。
原文摘要 · Abstract (English)
While modern text-to-image (T2I) models excel at generating images from intricate prompts, they struggle to capture the key details when the inputs are descriptive paragraphs. This limitation stems from the prevalence of concise captions that shape their training distributions. Existing methods attempt to bridge this gap by either fine-tuning T2I models on long prompts, which generalizes poorly to longer lengths; or by projecting the oversize inputs into normal-prompt space and compromising fidelity. We propose Prompt Refraction for Intricate Scene Modeling (PRISM), a compositional approach that enables pre-trained T2I models to process long sequence inputs. PRISM uses a lightweight module to extract constituent representations from the long prompts. The T2I model makes independent noise predictions for each component, and their outputs are merged into a single denoising step using energy-based conjunction. We evaluate PRISM across a wide range of model architectures, showing comparable performances to models fine-tuned on the same training data. Furthermore, PRISM demonstrates superior generalization, outperforming baseline models by 7.4% on prompts over 500 tokens in a challenging public benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。