用加权提示技术让扩散模型零样本生成并自定义诗歌配图。
Do It Yourself (DIY): Modifying Images for Poems in a Zero-Shot Setting Using Weighted Prompt Manipulation
- 通过动态调节关键词权重，控制图像中语义元素的强弱。
- 在零样本条件下实现诗歌到图像的生成与可控修改。
- 适合对诗歌视觉化有定制需求的创作者或研究者。
诗歌是一种富有表现力的艺术形式，其解读因读者的情感、经历和文化背景而异。为响应这一特性，我们致力于在零样本设置下生成诗歌对应的图像，并允许观众根据自身需求进行修改。为此，我们提出一种新颖的加权提示操控（Weighted Prompt Manipulation, WPM）技术，该技术系统性地调整扩散模型中的注意力权重与文本嵌入。通过动态调节特定词汇的重要性，WPM增强了或抑制了其在最终生成图像中的影响，从而产生语义更丰富、上下文更准确的可视化结果。本方法结合了扩散模型与大型语言模型（如GPT）及现有诗歌数据集，构建了文学领域图像生成的全面且结构化的方法论。据我们所知，这是首次将加权提示操控应用于提升诗歌语言的视觉表达。
原文摘要 · Abstract (English)
Poetry is an expressive form of art that invites multiple interpretations, as readers often bring their own emotions, experiences, and cultural backgrounds into their understanding of a poem. Recognizing this, we aim to generate images for poems and improve these images in a zero-shot setting, enabling audiences to modify images as per their requirements. To achieve this, we introduce a novel Weighted Prompt Manipulation (WPM) technique, which systematically modifies attention weights and text embeddings within diffusion models. By dynamically adjusting the importance of specific words, WPM enhances or suppresses their influence in the final generated image, leading to semantically richer and more contextually accurate visualizations. Our approach exploits diffusion models and large language models (LLMs) such as GPT in conjunction with existing poetry datasets, ensuring a comprehensive and structured methodology for improved image generation in the literary domain. To the best of our knowledge, this is the first attempt at integrating weighted prompt manipulation for enhancing imagery in poetic language.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。