小模型中决策提前锁定，且无法修正。
What Is the Minimum Architecture for Prolepsis? Early Irrevocable Commitment Across Tasks in Small Transformers
- 发现变压器在早期就锁定决策，特定注意力头维持该决定。
- 决策在最后提示词处出现单点尖峰，仅16层内可搜索修正。
- 适用于研究模型内部机制或决策过程的读者。
当变压器何时做出决策，以及什么阻止其修正？我们提出‘预判’（prolepsis）：变压器在早期即锁定决策，特定任务注意力头维持该决定，且后续层无法纠正。在两个开源模型（Gemma 2 2B、Llama 3.2 1B）上复现Lindsey等（2025）的规划位点发现，提出五个问题：（Q1）规划信号对六种残差流方法不可见；其中仅有基于CLT的操控有效。（Q2）单点尖峰在最终提示符处重现，形状一致（Anthropic的定位为换行符，详见新增注释）。（Q3）特定注意力头将决策传递至输出，填补了归因图未覆盖的路径空白。（Q4）证据支持最多16层内存在搜索，之后即完成锁定，形成双模型假设。（Q5）事实回忆任务也呈现相同模式，但重复使用的规划头与事实前10名无重叠。该现象在测试的解码器模型中跨任务重复出现：模板共享，路由结构不同。所有实验均在单块消费级显卡（16 GB VRAM）上完成。
原文摘要 · Abstract (English)
When do transformers commit to a decision, and what prevents them from correcting it? We introduce prolepsis: a transformer commits early, task-specific attention heads sustain the commitment, and no layer corrects it. Replicating Lindsey et al.'s (2025) planning-site finding on open models (Gemma 2 2B, Llama 3.2 1B), we ask five questions. (Q1) Planning is invisible to six residual-stream methods; among those tested, only CLT-based steering succeeds. (Q2) The single-site spike replicates in shape, at the final prompt token (Anthropic's site is the newline; see the Note added). (Q3) Specific attention heads route the decision to the output, filling a gap flagged as invisible to attribution graphs. (Q4) The evidence is consistent with search within at most 16 layers and commitment beyond, a two-model hypothesis. (Q5) Factual recall shows the same motif at a different network depth, with zero overlap between recurring planning heads and the factual top-10. Prolepsis recurs across tasks in the decoder-only models tested: the template is shared, the routing substrates differ. All experiments run on a single consumer GPU (16 GB VRAM).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。