arXiv:2412.06771cs.AIcs.CV2024-12ICML被引 26

主动提问的AI图像生成助手,能减少用户反复修改提示词的麻烦。

Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty

  • 当不确定用户意图时,主动提问澄清
  • 在三个数据集上使对齐效果提升至少2倍
  • 适合需要精准图像生成的设计师和创作者

生成式AI模型的用户提示常不完整,导致用户意图与模型理解错位,需反复修改提示。本文研究文本到图像生成中的对齐问题,提出一种具备主动澄清能力的智能体原型,可通过界面(1)在不确定时主动提问,(2)以可编辑的信念图展示其对用户意图的不确定性。构建了简易原型,并设计了一种基于双智能体的自动化评估方法:一个拥有真实意图(图像),另一个尽量少提问实现对齐。在ImageInWords、COCO和自建的DesignBench(含强艺术与设计元素)三个数据集上测试,结果表明该智能体能提出有效问题,成功对齐意图,且VQAScore较标准T2I生成提升至少2倍。人类实验显示,至少90%用户认为该智能体及其信念图有助于工作流程。代码与DesignBench详见https://github.com/google-deepmind/proactive_t2i_agents。

原文摘要 · Abstract (English)

User prompts for generative AI models are often underspecified, leading to a misalignment between the user intent and models' understanding. As a result, users commonly have to painstakingly refine their prompts. We study this alignment problem in text-to-image (T2I) generation and propose a prototype for proactive T2I agents equipped with an interface to (1) actively ask clarification questions when uncertain, and (2) present their uncertainty about user intent as an understandable and editable belief graph. We build simple prototypes for such agents and propose a new scalable and automated evaluation approach using two agents, one with a ground truth intent (an image) while the other tries to ask as few questions as possible to align with the ground truth. We experiment over three image-text datasets: ImageInWords (Garg et al., 2024), COCO (Lin et al., 2014) and DesignBench, a benchmark we curated with strong artistic and design elements. Experiments over the three datasets demonstrate the proposed T2I agents' ability to ask informative questions and elicit crucial information to achieve successful alignment with at least 2 times higher VQAScore (Lin et al., 2024) than the standard T2I generation. Moreover, we conducted human studies and observed that at least 90% of human subjects found these agents and their belief graphs helpful for their T2I workflow, highlighting the effectiveness of our approach. Code and DesignBench can be found at https://github.com/google-deepmind/proactive_t2i_agents.

图像生成主动智能体提示优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。