用大模型动态优化提示词，让图文生成更贴合用户意图。
TextMatch: Enhancing Image-Text Consistency Through Multimodal Optimization
- 通过LLM和VQA评分机制评估图文一致性
- 迭代优化提示词，显著提升生成图像与文本匹配度
- 适合需要精准控制生成内容的研究者和开发者
文本到图像生成模型在创作图像方面表现优异，但在确保输出与提示词的一致性上存在挑战。本文提出TextMatch框架，利用多模态优化解决图文生成中的语义偏差问题。该方法基于大语言模型（LLM）和视觉问答（VQA）模型构建评分策略，评估提示词与生成图像间的语义一致性。结合多模态上下文学习与思维链推理，实现提示词的动态迭代优化，使生成图像更准确反映用户意图，提升保真度与相关性。大量实验表明，TextMatch在多个基准测试中显著改善了图文一致性，为提升文本到图像生成模型能力提供了可靠方案。代码已公开于https://anonymous.4open.science/r/TextMatch-F55C/。
原文摘要 · Abstract (English)
Text-to-image generative models excel in creating images from text but struggle with ensuring alignment and consistency between outputs and prompts. This paper introduces TextMatch, a novel framework that leverages multimodal optimization to address image-text discrepancies in text-to-image (T2I) generation and editing. TextMatch employs a scoring strategy powered by large language models (LLMs) and visual question-answering (VQA) models to evaluate semantic consistency between prompts and generated images. By integrating multimodal in-context learning and chain of thought reasoning, our method dynamically refines prompts through iterative optimization. This process ensures that the generated images better capture user intent of, resulting in higher fidelity and relevance. Extensive experiments demonstrate that TextMatch significantly improves text-image consistency across multiple benchmarks, establishing a reliable framework for advancing the capabilities of text-to-image generative models. Our code is available at https://anonymous.4open.science/r/TextMatch-F55C/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。