TTT通过键值绑定实现线性注意力，提升效率与性能。
Test-Time Training with KV Binding Is Secretly Linear Attention
- 将TTT重构为可学习的线性注意力机制。
- 实现并行化计算，速度更快且保持性能不变。
- 解释了以往难以理解的模型行为，适合关注高效推理的读者。
测试时训练（TTT）中使用键值绑定作为序列建模层，通常被理解为一种在线元学习，能在测试时记忆键值映射。然而，我们的分析揭示了多个与此记忆机制相矛盾的现象。基于这些发现，我们重新审视了TTT的公式，表明一类广泛的TTT架构可表示为一种可学习的线性注意力算子。这一视角不仅解释了先前令人困惑的模型行为，还带来了多项实际优势：支持有原则的结构简化，允许完全并行的实现，在保持性能的同时提升效率，并能系统地将多种TTT变体归约为标准线性注意力形式。总体而言，我们的结果将TTT重新诠释为具有增强表征能力的可学习线性注意力，而非测试时的记忆机制。
原文摘要 · Abstract (English)
Test-time training (TTT) with KV binding as sequence modeling layer is commonly interpreted as a form of online meta-learning that memorizes a key-value mapping at test time. However, our analysis reveals multiple phenomena that contradict this memorization-based interpretation. Motivated by these findings, we revisit the formulation of TTT and show that a broad class of TTT architectures can be expressed as a form of learned linear attention operator. Beyond explaining previously puzzling model behaviors, this perspective yields multiple practical benefits: it enables principled architectural simplifications, admits fully parallel formulations that preserve performance while improving efficiency, and provides a systematic reduction of diverse TTT variants to a standard linear attention form. Overall, our results reframe TTT not as test-time memorization, but as learned linear attention with enhanced representational capacity. Project page: https://research.nvidia.com/labs/sil/projects/tttla/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。