分析多语言文生图模型重表面轻语义的问题
SoS: Analysis of Surface over Semantics in Multilingual Text-To-Image Generation
- 构建跨171种文化身份的多语言提示数据集
- 发现7个模型中6个在至少2种语言中出现严重表面倾向
- 表面倾向随文本编码层加深而加剧，常导致刻板视觉呈现
文生图（T2I）模型在全球用户中日益普及。然而，已有研究指出其对特定输入语言高度敏感——当面对非英语（即同一提示的不同表面形式）时，模型常生成文化刻板图像，优先考虑表面形式而非语义。我们首次系统分析了这种表面重于语义（SoS）现象。为此，我们创建了一个覆盖171种文化身份、翻译成14种语言的提示集，并用于测试7个T2I模型。为量化不同模型、语言和文化间的SoS倾向，我们提出一种新度量方法，并分析其视觉表现。结果显示，除一个模型外，其余均在至少两种语言中表现出强烈表面倾向，且该倾向随T2I文本编码器层级加深而增强。此外，这些表面倾向常与刻板视觉表征相关。
原文摘要 · Abstract (English)
Text-to-image (T2I) models are increasingly employed by users worldwide. However, prior research has pointed to the high sensitivity of T2I towards particular input languages - when faced with languages other than English (i.e., different surface forms of the same prompt), T2I models often produce culturally stereotypical depictions, prioritizing the surface over the prompt's semantics. Yet a comprehensive analysis of this behavior, which we dub Surface-over-Semantics (SoS), is missing. We present the first analysis of T2I models' SoS tendencies. To this end, we create a set of prompts covering 171 cultural identities, translated into 14 languages, and use it to prompt seven T2I models. To quantify SoS tendencies across models, languages, and cultures, we introduce a novel measure and analyze how the tendencies we identify manifest visually. We show that all but one model exhibit strong surface-level tendency in at least two languages, with this effect intensifying across the layers of T2I text encoders. Moreover, these surface tendencies frequently correlate with stereotypical visual depictions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。