通过精炼实体提示提升文本生成图像的准确性
TextTIGER: Text-based Intelligent Generation with Entity Prompt Refinement for Text-to-Image Generation
- 用外部信息扩展实体描述并大模型总结,避免输入过长
- 在多个评估指标上优于仅用原始提示的方法
- 适合需要精准生成特定实体图像的研究者使用
生成包含特定实体的图像时,模型需保留尽可能多的实体相关知识。然而实体数量几乎无限且持续涌现,完全记忆所有实体不现实。为此,本文提出文本驱动的智能生成方法TextTIGER,通过引入外部信息增强提示中的实体描述,并利用大语言模型对扩展内容进行摘要,从而防止因输入过长导致性能下降。为评估该方法,我们构建了一个新数据集,包含图文对、详细描述及实体列表。在多个图像生成模型上的实验表明,与仅使用原始标题提示相比,TextTIGER在常用评估指标上显著提升生成效果。此外,通过多模态大模型作为评判器(MLLM-as-a-judge),其结果与人类评价高度相关,证明本方法始终获得更高评分,验证了其有效性。这些结果表明,强化实体描述、合理摘要并优化提示长度,可显著提升图像生成性能。代码与数据集将在论文录用后公开。
原文摘要 · Abstract (English)
When generating images from prompts that include specific entities, the model must retain as much entity-specific knowledge as possible. However, the number of entities is almost countless, and new entities emerge; memorizing all of them completely is not realistic. To bridge this gap, our work proposes Text-based Intelligent Generation with Entity Prompt Refinement (TextTIGER). TextTIGER strengthens knowledge about entities that appear in the prompt by augmenting external information and then summarizes the expanded descriptions with large language models, preventing performance degradation that arises from excessively long inputs. To evaluate our method, we construct a new dataset consisting of captions, images, detailed descriptions, and lists of entities. Experiments with multiple image generation models show that TextTIGER improves image generation performance on widely used evaluation metrics compared with prompts that use captions alone. In addition, using Multimodal LLM (MLLM)-as-a-judge, which shows a strong correlation with human evaluation, we demonstrate that our method consistently achieves higher scores, which underscores its effectiveness. These results show that strengthening entity-related descriptions, summarizing them, and refining prompts to an appropriate length leads to substantial improvements in image generation performance. We will release the created dataset and code upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。