修正测试时训练的检索机制缺陷，发现其无法真正实现精准记忆。
SR-TTT Does Not Learn Retrieval: A Correction and Mechanistic Post-Mortem of Surprisal-Aware Residual Test-Time Training
- 通过修正评估错误，发现原方法依赖输入中已存在的答案而非真正检索
- 实际存储能力受限，早期关键信息几乎无法被捕捉（0-1%命中率）
- 即使改进读取机制，仍无法突破近似0.06纳特的信息提取上限
测试时训练（TTT）语言模型在推理时用快速权重替代键值缓存，实现O(1)内存但难以完成精确召回任务。之前提出的SR-TTT通过将高意外性标记路由至稀疏的精确注意力残差缓存，在‘针堆找针’任务中报告显著提升。我们发现这些提升是评估漏洞所致：损失与指标读取的是答案位置的logits而非前一位置，导致模型仅需复制输入中已可见的答案即可获得100%准确率；此外，缓存非因果地访问了未来标记，包括答案本身。我们发布了修正后的实现及启动因果自检，重新检验假设。结果表明，原始假说不成立，失败可分解为两个独立瓶颈：存储方面，意外性门控存在系统性位置偏差——需要预热期才能使目标标记变得相对意外，因此长上下文最需要记忆的早期标记存储率极低（深度0.1时为0-1%）；读取方面，即使使用理想存储和可训练读取投影，每槽注意力监督使定位质量提升2.5倍（0.06→0.15），但标记准确率无统计变化，检索仅提取约0.06纳特（总2.30纳特），且内容定位无法解决内容高度相似的有序槽问题。所有2,250对实验条件下精确匹配率始终为0%。我们撤回版本1的主张，提供修正代码库、诊断协议与负结果，作为对意外性门控记忆架构的警示参考。
原文摘要 · Abstract (English)
Test-Time Training (TTT) language models replace the KV-cache with fast weights updated during inference, achieving O(1) memory but suffering catastrophic failure on exact-recall tasks. Version 1 of this work proposed SR-TTT, which routes high-surprisal tokens to a sparse exact-attention Residual Cache, and reported large Needle-in-a-Haystack gains. We show those gains were evaluation artifacts: the loss and metric read logits at the answer positions rather than one position earlier, training both models to copy an answer already visible in their input (a model trained on retrieval-impossible data reaches 100% accuracy under the flawed metric); additionally, the cache attended non-causally over future tokens, including the answer itself. We release a corrected implementation with startup causality self-tests, then ask whether the SR-TTT hypothesis survives correction. It does not, and the failure decomposes into two independent, separately measured bottlenecks. Storage: surprisal gating is systematically position-biased - the TTT reconstruction loss requires burn-in before a needle becomes relatively surprising, so early-context needles are stored at near-zero rates (0-1% containment at depth 0.1) exactly where long-context memory matters most. Addressing: with storage solved by an oracle and with new trainable read-time projections, per-slot attention supervision raises addressing mass 2.5x (0.06 -> 0.15) yet token accuracy is statistically unchanged, and retrieval extracts only approx. 0.06 nats of the 2.30-nat needle; position-free content addressing cannot resolve ordered slots whose contents are near-interchangeable. Exact match remains 0% in all 2,250 paired trials across all corrected conditions. We retract the claims of v1 and offer the corrected codebase, diagnostic protocol, and negative results as a cautionary reference for surprise-gated memory architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。