arXiv:2608.24460cs.CL2026-08

通过构造特殊语料,揭示模型在上下文冲突时如何依赖时间与罕见性线索。

Shortcut Before Circuit: Document Statistics Time In-Context Conflict Resolution

  • 设计合成数据让时间与罕见性线索完全重合,隔离因果影响。
  • 75次实验准确率均超0.999,但内部机制差异显著,部分单元符号占比差超0.3。
  • 发现模型先依赖位置捷径,后突破,其机制可探测但不唯一,适合研究模型决策逻辑者阅读。

当上下文对同一事实给出两个不同值时,模型会依据某一提示(如新近性、重复性、位置)做出选择,但真实数据中这些线索通常一致,难以区分模型真正依赖的机制。我们训练了一个2600万参数的Transformer,在一个合成语言中使新近性与罕见性完全重合,并通过最小因果编辑反转其中一个线索,同时保持真相、词元数量和答案位置不变。所有75次运行的准确率均不低于0.999,包括那些简单启发式失效的情况,因此常规评估无法区分它们。但在干预下,每个单元的读出结果不具可复现性:25个单元中有13个在三次种子下符号占比差异超过0.3,最大偏差达0.879,标准误为0.025。该现象可被构造解释——共现规则使目标函数对二者无偏好,而方差由每组比较释放的优化量决定。可复现的是时间模式:脱离位置捷径具有闭式上界,且单调依赖冗余度。在逃脱前探测,75次中有32次归因符号反转,但准确率不变;电路形成需门控机制,但非充分条件。语料决定机制何时出现,而非具体是哪个机制——这为机制归因何时可用提供了判据,而我们的构造使不可用情形变得精确。

原文摘要 · Abstract (English)

When a context asserts two values for one fact, a model commits to a cue -- recency, repetition, position -- but natural data rarely makes these disagree, so behavior cannot reveal which. We train 26M-parameter transformers on a synthetic language where recency and rarity are exactly coextensive, and separate them with a minimal causal edit that inverts one cue while holding the truth, token count and answer position fixed. All 75 runs reach accuracy >= 0.999, including where the trivial heuristic fails, so no held-in evaluation distinguishes them. Under intervention the per-cell readout does not replicate: 13 of 25 cells differ by more than 0.3 in sign fraction across three seeds, the largest by 0.879 against a standard error of 0.025. The construction predicts this -- coextensive rules leave the objective indifferent between them -- and the variance is ordered by how much of the optimization each comparison releases. What replicates is timing: escape from a positional shortcut with a closed-form ceiling, monotone in redundancy. Probed before that escape, attribution reverses sign in 32 of 75 runs at unchanged accuracy, and gating on circuit formation is necessary but not sufficient. The corpus fixes when a mechanism appears, not which one -- a criterion for when mechanistic attribution to data is available at all, and our construction makes the unavailable case exact.

模型机制上下文推理可解释性因果分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。