让循环模型在训练长度外仍能有效推理，突破传统时序建模瓶颈。
Learning Length-Extrapolatable Recurrent Models
- 通过稳定状态信用信号，改进反向传播中的梯度传递机制
- 在真实数据与合成任务上实现最高128倍于训练长度的性能提升
- 适用于长序列建模，尤其适合需要超长上下文的场景
循环模型为长上下文建模提供了自然路径，但通过时间反向传播（BPTT）训练的模型通常在训练范围外失效。经典分析强调沿时间路径的梯度消失或爆炸，然而密集的逐标记损失仍可训练共享的循环规则，即使存在严重衰减，表明衰减本身并非学习失败的决定因素。本文转而研究状态信用：未来损失在参数更新前回传至早期状态的信号。为此，提出时间信用稳定（CST），在反向传播中局部缩放状态信用信号以稳定其范数，不旋转待修正分量，且保持前向计算不变。由于合成任务与真实数据呈现不同的信用动态，将CST分别适配两类场景。在两种设置下，CST均显著提升训练长度之外的性能，最大增益达128倍训练长度。
原文摘要 · Abstract (English)
Recurrent models provide a natural path to long-context modeling, yet models trained with backpropagation through time (BPTT) often fail beyond their training horizon. Classical analyses emphasize gradients that vanish or explode along temporal paths. However, dense per-token losses can still train a shared recurrent rule despite severe decay, showing that decay alone does not determine whether learning fails. We instead study state credit: the signal through which future losses reach earlier recurrent states before contributing to parameter updates. Accordingly, we intervene directly on state credit and propose Credit Stabilization through Time (CST). During backward propagation, CST locally rescales the state-credit signal to stabilize its norm without rotating the component being corrected, while leaving the forward computation unchanged. Because controlled synthetic tasks and real data exhibit different credit dynamics, we specialize CST to each regime. In both settings, CST improves performance beyond the training horizon, with gains observed at up to 128x the training length.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。