arXiv:2507.23313cs.CV2025-07被引 1

分析扩散模型如何理解艺术提示中的内容与风格区分。

The Cow of Rembrandt - Analyzing Artistic Prompt Interpretation in Text-to-Image Models

  • 通过注意力热图定位提示词对图像区域的影响。
  • 内容词多影响物体区域,风格词多影响背景纹理。
  • 揭示模型在无监督下自发形成内容风格分离机制。

文本到图像扩散模型通过学习数十亿张图像(包括流行艺术品)展现出生成艺术内容的惊人能力。然而,这些模型如何内部表征绘画中的内容与风格概念仍不明确。传统计算机视觉认为内容与风格正交,但扩散模型训练时并未获得此类显式指导。本文研究基于Transformer的文本到图像扩散模型在生成艺术品时如何编码内容与风格概念。我们利用交叉注意力热图将生成图像的像素归因于特定提示词,从而分离出由内容描述词和风格描述词影响的图像区域。结果显示,模型在不同艺术提示和风格请求下表现出不同程度的内容-风格分离:多数情况下,内容词主要影响物体相关区域,而风格词影响背景和纹理区域,暗示模型自发形成了内容与风格的区分理解。这些发现增进了我们对大规模生成模型在无监督条件下表征复杂艺术概念的理解。代码、数据集及可视化工具已开源:https://github.com/umilISLab/artistic-prompt-interpretation。

原文摘要 · Abstract (English)

Text-to-image diffusion models have demonstrated remarkable capabilities in generating artistic content by learning from billions of images, including popular artworks. However, the fundamental question of how these models internally represent concepts, such as content and style in paintings, remains unexplored. Traditional computer vision assumes content and style are orthogonal, but diffusion models receive no explicit guidance about this distinction during training. In this work, we investigate how transformer-based text-to-image diffusion models encode content and style concepts when generating artworks. We leverage cross-attention heatmaps to attribute pixels in generated images to specific prompt tokens, enabling us to isolate image regions influenced by content-describing versus style-describing tokens. Our findings reveal that diffusion models demonstrate varying degrees of content-style separation depending on the specific artistic prompt and style requested. In many cases, content tokens primarily influence object-related regions while style tokens affect background and texture areas, suggesting an emergent understanding of the content-style distinction. These insights contribute to our understanding of how large-scale generative models internally represent complex artistic concepts without explicit supervision. We share the code and dataset, together with an exploratory tool for visualizing attention maps at https://github.com/umilISLab/artistic-prompt-interpretation.

图像生成注意力分析艺术理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。