arXiv:2506.23138cs.CV2025-06被引 3

让文字生成图像更懂用户意图,自动优化提示词语义一致性。

VisualPrompter: Semantic-Aware Prompt Optimization with Visual Feedback for Text-to-Image Synthesis

  • 通过视觉反馈识别生成图缺了什么概念,逐原子级修正提示词
  • 在多个评测集上达到新最好效果,图文对齐度显著提升
  • 无需训练、即插即用,适配各类文生图模型

用户提供的提示词与模型偏好提示词之间存在显著差距,制约高质量图像生成,亟需提示词工程。现有方法虽能提升图像风格与美感,但常忽视生成图像与用户描述的语义对齐,导致画面美观却内容不符。本文提出VisualPrompter,一种无需训练的提示词优化框架,可将用户输入转化为模型偏好的表达。该框架包含自动自我反思模块,识别生成图像中缺失的概念,并通过目标特定的细粒度提示词优化机制进行修正。通过拆解提示词,在原子语义层面引入新元素并重新组合,确保优化过程中的语义一致性和完整性。大量实验表明,VisualPrompter在多个图文对齐评估基准上取得新最优性能。此外,其即插即用设计使其高度兼容各类生成模型。代码已公开于https://github.com/teheperinko541/VisualPrompter。

原文摘要 · Abstract (English)

The notable gap between user-provided and model-preferred prompts poses a significant challenge for generating high-quality images with text-to-image models, compelling the need for prompt engineering. Current studies on prompt engineering can effectively enhance the style and aesthetics of generated images. However, they often neglect the semantic alignment between generated images and user descriptions, resulting in visually appealing but content-wise unsatisfying outputs. In this work, we propose VisualPrompter, a novel training-free prompt engineering framework that refines user inputs to model-preferred sentences. VisualPrompter utilizes an automatic self-reflection module that identifies absent concepts in the generated images, followed by a target-specific prompt optimization mechanism that revises the prompts in a fine-grained manner. By deconstructing prompts, introducing new elements at the atomic semantic level, and then reassembling them, our framework is able to maintain semantic consistency and integrity throughout the optimization process. Extensive experiments demonstrate the effectiveness of VisualPrompter, which achieves new state-of-the-art performance on multiple benchmarks for text-image alignment evaluation. Additionally, our framework features a plug-and-play design, making it highly adaptable to various generative models. Our code is available at https://github.com/teheperinko541/VisualPrompter.

文生图提示词优化语义对齐视觉反馈

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。