研究文字样式如何影响大模型对图像中文本概念的属性描述
Revealing the Impact of Visual Text Style on Attribute-based Descriptions Produced by Large Visual Language Models
- 对比可读性与装饰性文字风格对模型描述的影响
- 即使正确识别概念，风格仍改变属性描述结果
- 揭示视觉风格泄露问题，适合关注模型公平性的研究者
当考虑文本的视觉风格时，字体、颜色和大小存在广泛差异。然而，阅读一个词时，其含义独立于书写或呈现的风格。本文研究了文字在图像中的视觉风格是否以及如何影响大视觉语言模型（LVLM）对所指概念的属性描述。具体而言，我们考察了功能型风格（如黑色无衬线体，注重可读性）与装饰型风格（如彩色手写体，注重展示性）对LVLM描述概念属性的影响。实验设定在模型能正确识别视觉文本所指概念的前提下，此时文本风格本不应影响属性描述。结果显示，即便概念被正确识别，文本风格仍显著影响模型的属性描述。研究揭示了视觉风格信息非平凡地渗入语义推理过程，为基于LVLM的多媒体系统提出了风格感知评估与缓解的必要性。
原文摘要 · Abstract (English)
When the visual style of text is considered, a wide variety can be observed in font, color, and size. However, when a word is read, its meaning is independent of the style in which it has been written or rendered. In this paper, we investigate whether, and how, the style in which a word is visualized in an image impacts the description that a Large Visual Language Model (LVLM) provides for the concept to which that word refers. Specifically, we investigate how functional text styles (readability-oriented, e.g., black sans-serif) versus decorative styles (display-oriented, e.g., colored cursive/script) affect LVLMs' descriptions of a concept in terms of the attributes of that concept. Our experiments study the situation in which the LVLM is able to correctly identify the concept referred to by a visual text, i.e., by a word or words rendered as an image, and in which the visual text style should not influence the attribute-based description that the LVLM produces. Our experimental results reveal that even when the concept is correctly identified, text style influences the model's attribute-based descriptions of the concept. Our findings demonstrate non-trivial style leakage from text style into semantic inference and motivate style-aware evaluation and mitigation for LVLM-based multimedia systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。