发现大模型会因上下文句子而提高其出现概率，且可通过关闭部分注意力头抑制。
Sentence-Level Contextual Entrainment in Large Language Models
- 用句子级平均对数概率代替单个词概率，研究上下文句式影响
- 26个模型验证：即使虚构句子也会在推理时概率显著上升
- 仅关闭2%-4%注意力头即可有效缓解，不影响模型性能
上下文同步（contextual entrainment）是大语言模型中一种新发现的现象，指模型倾向于为上下文中出现的词分配更高的概率。本文将该现象从词级拓展至句级，通过分析句子的逐词平均对数概率，而非单个词的概率。我们在来自七个模型家族的26个大模型上，基于两个数据集（涵盖主观与客观任务）进行了研究。结果表明，句级上下文同步确实存在：即使提示中的句子为反事实陈述，其在推理过程中概率也会显著提升。随着模型规模增大，上下文同步现象逐渐减弱。我们还发现，这一现象受2%至4%的注意力头控制；关闭这些注意力头可有效缓解上下文同步，且不损害模型性能。
原文摘要 · Abstract (English)
Contextual entrainment, which is a newly discovered phenomenon in large language models (LLMs), refers to the tendency of a model to assign higher probabilities to tokens that appear in its context. In this work, we extend this phenomenon from the token level to the sentence level by examining the per-token mean log-probability of a sentence instead of the probabilities of individual tokens. We investigate sentence-level contextual entrainment across 26 LLMs from seven families and two datasets, which cover both subjective and objective tasks. We find that sentence-level contextual entrainment exists. This means that the sentences in the prompt (even if they are counterfactual statements) can significantly increase their probability during model inference time. As the model size increases, contextual entrainment gradually decreases. We also find that contextual entrainment is controlled by 2% to 4% of the attention heads. Turning off these attention heads can effectively mitigate contextual entrainment without hurting the model's performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。