大模型越训练越会忽略假信息，但也越容易照搬无关词，表现两极分化。
Better and Worse with Scale: How Contextual Entrainment Diverges with Model Size
- 通过幂律规律量化模型对上下文的依赖程度（即‘纠缠’行为）
- 大模型对语义上下文依赖下降，对非语义内容依赖上升，且差距达4倍和2倍
- 适合关注模型可解释性与上下文敏感性的研究者阅读
更大的语言模型在处理上下文信息时表现出矛盾现象：一方面更善于忽略错误陈述，另一方面却更容易受无关词干扰。本文首次建立上下文纠缠（contextual entrainment）的缩放定律，分析了Cerebras-GPT（111M-13B）和Pythia（410M-12B）模型家族。结果表明，纠缠行为遵循可预测的幂律缩放，但趋势因上下文类型而异：语义上下文下的纠缠随规模减小，非语义上下文则相反。具体而言，最大模型比最小模型对反事实误导信息的抵抗能力高出四倍，同时对任意词语的复制倾向增加两倍。这种跨模型族一致的分化表明，语义过滤与机械复写是功能上独立、且呈相反趋势的行为，说明规模扩展并未解决上下文敏感性问题，而是重塑了它。
原文摘要 · Abstract (English)
Larger language models become simultaneously better and worse at handling contextual information -- better at ignoring false claims, worse at ignoring irrelevant tokens. We formalize this apparent paradox through the first scaling laws for contextual entrainment, the tendency of models to favor tokens that appeared in context regardless of relevance. Analyzing the Cerebras-GPT (111M-13B) and Pythia (410M-12B) model families, we find entrainment follows predictable power-law scaling, but with opposite trends depending on context type: semantic contexts show decreasing entrainment with scale, while non-semantic contexts show increasing entrainment. Concretely, the largest models are four times more resistant to counterfactual misinformation than the smallest, yet simultaneously twice as prone to copying arbitrary tokens. These diverging trends, which replicate across model families, suggest that semantic filtering and mechanical copying are functionally distinct behaviors that scale in opposition -- scaling alone does not resolve context sensitivity, it reshapes it.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。