重复词的位置影响模型预测,远距离重复反而降低准确率。
When More Becomes Less: Position-Dependent Repetition Effects in Language Models

- 设计两种探针,固定重复词块,只变读取位置。
- 远距离重复时预测概率先升后降,形成倒U型曲线。
- 该现象在13种模型、4种语言中均显著,适合研究模型注意力机制。
基于填空的探测实验通常假设目标词出现次数越多,其预测概率越高,且不受读取位置影响。本文通过双探针设计发现这一假设不成立:当读取位置紧邻重复词块时,预测概率随重复次数N增加而上升并趋于饱和;而当读取位置位于新句框架内时,预测概率呈现倒U型变化——先升后降。在13个开源编码器与解码器模型中,倒U型趋势显著,95%置信区间排除零值;在西班牙语、中文、德语、法语共42个跨语言测试中全部复现。六条件因果消融分析表明,该效应源于精确词汇重复,而非长度、泛化冗余或语义邻近暴露。帧-语用控制排除了读取帧本身造成的伪影。内部分析显示,在因果语言模型中,每个目标词的注意力下降,但整个重复块获得的总注意力预算上升,而掩码语言模型中无此现象。因此,重复次数与读取位置不可分离。
原文摘要 · Abstract (English)
Cloze-style probes that vary how often a target token appears implicitly assume that more copies of a target affect prediction the same way regardless of where the readout slot sits. We show this assumption fails. Our two-probe design holds a repeated-target prefix fixed and varies only the readout position: the adjacent probe places the slot immediately after the repeated block; the displaced probe places it inside a fresh sentence frame. Adjacent repetition behaves as priming intuition predicts: $P(\text{target})$ climbs with $N$ and plateaus. Displaced repetition produces an inverted-U: $P(\text{target})$ rises to an early peak and then declines as more copies are added. The displaced inverted-U shows a per-word drop with bootstrap CI excluding zero in all 13 open-access encoder and decoder models we test, and replicates across Spanish, Chinese, German, and French in 42 of 42 multilingual cells. A six-condition causal ablation isolates the effect to exact lexical repetition rather than length, generic redundancy, or semantic-neighbour exposure. A frame-pragmatics control rules out an artefact of the readout frame. Internally, per-target-token attention falls with $N$ while the total budget assigned to the repeated block grows in causal LMs but not in the masked LM we probe. Probes that vary repetition count cannot treat the readout position as orthogonal to what they measure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。