arXiv:2608.03711cs.CVcs.CL2026-08中稿 · ECCV

发现大模型会因大小写分布自动聚焦关键文本,像人眼一样被大写字母吸引。

Attention is Case-Sensitive

论文配图:Attention is Case-Sensitive
图 1 · 摘自论文原文
  • 通过13个模型实验证明,大写字符能自然引导注意力分配。
  • 大写聚焦虽明显但不提升准确率,高混乱度下甚至降低性能。
  • 推理模型的思考阶段可减弱对大小写的敏感,适合设计提示策略的人参考。

在人类视觉感知中,大写字母是吸引注意力的自然线索。本文系统研究发现,大型语言模型(LLM)也具备类似特性:字母大小写会影响内部注意力分配。通过对13个模型(9个LLM和4个视觉-语言模型,VLM)及多种分词方案的分析,我们发现将目标信息以交替或大写形式置于小写上下文中,能显著集中注意力于这些片段。该效应在所有非推理类模型中普遍存在。我们将其视为预训练变压器中未被充分探索的潜在属性,而非显式方法。研究揭示了注意力与性能间的根本分歧:尽管大小写效应能稳健地转移注意力,但其对下游任务准确率的影响并不显著——注意力增强不等于性能提升,在高熵语境如交替大小写中甚至会下降。进一步发现,推理模型的“思考”阶段作为语义缓冲区,可缓解文本中的排版敏感性。扩展至VLM时,该效应部分迁移:相同提示侧的大小写变化会重构跨模态注意力,主要表现为图像注意力整体向文本提示偏移,次要表现为剩余视觉注意力集中在目标区域。通过将大小写作为无需模型访问或微调的零样本注意力引导机制,本研究为预训练模型如何内化排版强调提供了新基础认知。

原文摘要 · Abstract (English)

In human visual perception, uppercase lettering serves as a natural salience cue that captures attention within lowercase text. In this paper, we present a systematic empirical characterization study revealing that Large Language Models (LLMs) exhibit an analogous property: letter casing modulates internal attention allocation. Through analysis across 13 models, nine LLMs and four Vision-Language Models (VLMs), with diverse tokenization schemes, we show that formatting target information in alternating or uppercase against a lowercase context concentrates attention on those textual spans. In text this effect is universal, holding across every evaluated non-reasoning model. We frame it as a previously under-explored latent property of pretrained transformers rather than a prescriptive method. Our investigation reveals a central attention-performance divergence: while this "casing effect" robustly shifts attention, its impact on downstream accuracy is non-trivial, increased concentration does not inherently improve task accuracy and, in high-entropy contexts like alternating case, can degrade it. We further identify a boundary condition: the deliberative "thinking" phase in reasoning models acts as a semantic buffer that mitigates typographic sensitivity in text. Extending the study to VLMs, we find the effect transfers partially: the same prompt-side casing reorganizes cross-modal attention along two coupled axes, predominantly a macroscopic disengagement from the image toward the text prompt, and secondarily a concentration of the residual visual attention on the target region. By isolating casing as a zero-shot mechanism for attention steering that requires no model access or fine-tuning, we provide a new foundational understanding of how pretraining internalizes typographic emphasis.

注意力机制大模型提示工程多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。