测试视觉提示干扰下模型对颜色的误判，发现多数模型会受文字内容误导。
What Color Is the Text? A Benchmark for Hallucination Induced by Image-Embedded Prompt
- 设计嵌入式斯特鲁普任务，用文字颜色与真实颜色冲突测试模型
- 仅6.3%准确识别59种颜色，但21.6%出现文字干扰误答
- 文字可读性越强，模型越容易被误导，适合评估多模态模型可靠性
我们提出Embedded Stroop，一种用于测量多模态大模型中图像嵌入提示干扰的受控诊断范式，其中问题直接渲染在视觉输入中。基于涵盖59种细粒度颜色的What-Color-Is-the-Text（WCIT）基准，评估了16个开源及专有模型。为区分语义捕获失败与一般颜色命名错误，将模型输出分解为准确率、斯特鲁普幻觉率（SHR；回答嵌入文字而非真实文本颜色）和其他错误率，并通过置换检验验证，64种条件中有56种在FDR校正后仍显著。尽管在59色词汇下的精确颜色准确率为6.3%，但映射到11类基本颜色时，模型仍保持38.4%的粗粒度颜色感知能力，同时表现出显著的斯特鲁普幻觉率（21.6%）。在标准设置中正确命名的颜色上进行条件分析，进一步确认该效应，合并后的条件斯特鲁普幻觉率超过50%。遮蔽或翻转嵌入文字可降低斯特鲁普幻觉，表明语义可读性可能主导多模态大模型的视觉颜色感知。
原文摘要 · Abstract (English)
We introduce Embedded Stroop, a controlled diagnostic paradigm for measuring image-embedded prompt interference in Multimodal Large Language Models (MLLMs), where the query is rendered directly inside the visual input. Using the What-Color-Is-the-Text (WCIT) benchmark, which covers 59 fine-grained colors under Standard, Flipped, and Masked variants, we evaluate 16 proprietary and open-source models. To distinguish semantic capture from general color-naming failure, we decompose model responses into Accuracy, Stroop Hallucination Rate (SHR; answering the embedded word rather than the true text color), and Other Error Rate, and validate the effect with permutation tests, with 56 of 64 conditions remaining significant after FDR correction. Although exact color accuracy is low (6.3%) under the 59-color vocabulary, mapping predictions to 11 basic color families shows that models retain coarse color perception (38.4%) while still exhibiting a substantial SHR (21.6%). A conditional analysis restricted to colors correctly named in the Standard setting further confirms the effect, with pooled conditional SHR exceeding 50\%. Masking or flipping the embedded text reduces Stroop hallucinations, suggesting that semantic legibility can dominate visual color perception in MLLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。