arXiv:2608.07110cs.LGcs.CL2026-08被引 1

将测试时训练拆解为可组合模块,提升方法设计与性能分析效率。

Modular TTT: Rethinking Test-Time Training as Composable Modules

论文配图:Modular TTT: Rethinking Test-Time Training as Composable Modules
图 1 · 摘自论文原文
  • 把快权重更新拆成可配置的图结构模块,支持灵活组合
  • 小学习率初始化和权重衰减能显著提升性能
  • 适合想快速实验新测试时训练策略的研究者

测试时训练(TTT)将序列建模视为在线学习问题,通过内部学习规则快速更新快权重。现有方法通常硬编码每个变体,难以设计新方法或分离组件作用。为此,我们提出模块化TTT框架,将内层学习器表示为有向无环图,显式暴露快权重网络、损失函数、学习率、权重衰减和归一化等设计维度。该框架自动将基础的前向训练视图、反向训练视图和因果查询视图规则组合成完整的图级TTT计算流程,包括快权重状态转移。利用此框架,我们系统性地消融了各组件,发现小学习率初始化、权重衰减及单层非线性有助于性能提升,而MSE与内积损失表现相近。更深的快权重网络和归一化会因激活值过大反而损害性能,残差连接与门控机制则几乎无显著收益。基于这些发现,我们在1000亿标记符上训练出4.1亿与14.5亿参数的最优变体,其训练损失与基准性能可媲美Gated DeltaNet。

原文摘要 · Abstract (English)

Test-time training (TTT) views sequence modeling as an online learning problem in which fast weights are updated by an internal learning rule. Despite the growing number of TTT variants, existing approaches typically hard-code each variant separately, which makes it difficult to design new TTT methods and to isolate the role of each component. To address this, we propose Modular TTT, a framework that represents the inner learner as a directed acyclic graph and exposes the fast-weight network, loss function, learning rate, weight decay, and normalization as explicit design dimensions. Modular TTT automatically composes primitive-level train-view forward, train-view backward, and causal query-view rules into the full graph-level TTT computation, including the fast-weight state transition. Using Modular TTT, we systematically ablate the components of TTT and find that small learning-rate initialization, weight decay, and a single-layer nonlinearity improve performance, while MSE and inner-product losses perform similarly. Deeper fast-weight networks and normalization tend to hurt performance because they induce excessively large activations, while residual connections and gating provide little measurable benefit. Guided by these findings, we train the best resulting variant as 410M- and 1.45B-parameter models on 100B tokens, and observe training loss and benchmark performance comparable to Gated DeltaNet.

测试时训练模块化设计快权重序列建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。