arXiv:2505.22792cs.CV2025-05

让AI理解隐喻并生成对应图像,突破传统模型的字面误解

Rhetorical Text-to-Image Generation via Two-layer Diffusion Policy Optimization

  • 分两层优化生成过程:先拆解隐喻为逐步细化的子句,再沿去噪路径优化每步动作
  • 在多个评测中超越GPT-4o、Grok-3等顶尖模型,生成图像更贴合隐喻语义
  • 适合研究多模态理解、创意生成或需处理修辞表达的AI应用开发者

从修辞性语言生成图像仍是文本到图像模型的关键挑战。即使最先进(SOTA)的多模态大语言模型(MLLM)也难以基于修辞语言中的深层含义生成图像——尽管人类能轻易将此类内容映射到视觉表征。当前模型主要依赖物体级词嵌入对齐,导致比喻性表达被引导至字面视觉,忽略实际语义。为此,我们提出Rhet2Pix,将修辞性文本到图像生成建模为多步策略优化问题,引入两层马尔可夫决策过程(MDP)扩散模块。外层将输入提示逐步转化为更丰富的子句并执行图像生成动作,构建语义更丰富的视觉内容;内层通过折扣最终奖励,在扩散去噪轨迹上优化相邻动作对,缓解奖励稀疏问题。大量实验证明Rhet2Pix在修辞性文本到图像生成上的有效性,其性能超越GPT-4o、Grok-3等主流学术基线,在定性和定量评估中均表现优异。本工作代码与数据集已公开。

原文摘要 · Abstract (English)

Generating images from rhetorical languages remains a critical challenge for text-to-image models. Even state-of-the-art (SOTA) multimodal large language models (MLLM) fail to generate images based on the hidden meaning inherent in rhetorical language--despite such content being readily mappable to visual representations by humans. A key limitation is that current models emphasize object-level word embedding alignment, causing metaphorical expressions to steer image generation towards their literal visuals and overlook the intended semantic meaning. To address this, we propose Rhet2Pix, a framework that formulates rhetorical text-to-image generation as a multi-step policy optimization problem, incorporating a two-layer MDP diffusion module. In the outer layer, Rhet2Pix converts the input prompt into incrementally elaborated sub-sentences and executes corresponding image-generation actions, constructing semantically richer visuals. In the inner layer, Rhet2Pix mitigates reward sparsity during image generation by discounting the final reward and optimizing every adjacent action pair along the diffusion denoising trajectory. Extensive experiments demonstrate the effectiveness of Rhet2Pix in rhetorical text-to-image generation. Our model outperforms SOTA MLLMs such as GPT-4o, Grok-3 and leading academic baselines across both qualitative and quantitative evaluations. The code and dataset used in this work are publicly available.

文本生成修辞理解扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。