提出SANE方法解决极端长序列下递归模型的数值不稳定性问题。
SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models
- 在块边界使用自适应tanh压缩，保持块内并行结构。
- 1亿词元超长上下文后仍保持33.46~35.56的推理能力，基线已溢出。
- 揭示了数值稳定与推理能力间的权衡，适合长序列建模研究者。
Delta-Rule循环模型虽具固定状态大小、实现O(1)推理内存，但在极端上下文外推时可能失稳。通过追踪长达1亿词元序列的RWKV-7，我们实证发现其失效模式为：局部范数爆炸叠加于相对稀疏的基底之上，而非全局状态饱和。分析表明，持续衰减使弱更新项保持小值，而不均衡注入导致少数通道积累极端值。为此，我们提出状态异常中和（SANE），在块边界应用自适应tanh压缩，同时保留块内并行结构。在安全阈值范围（3 ≤ α ≤ 5）内，SANE在11个短上下文推理基准上与基线无显著差异；在超过训练长度24,000倍的1亿词元前缀后，SANE仍维持功能推理能力（33.46–35.56），而基线出现数值溢出。反之，过宽松阈值（α ≥ 8）虽数值稳定却完全丧失推理能力，表明数值稳定不等于功能可用，揭示状态压缩中的容量-稳定性权衡。
原文摘要 · Abstract (English)
Delta-Rule recurrent models maintain a fixed-size state, enabling $O(1)$ inference memory but potentially becoming unstable under extreme-context extrapolation. By tracking RWKV-7 over sequences of up to 100M tokens, we empirically identify a distinct failure pattern: \textbf{localized norm explosion atop a relatively sparse substrate}, rather than global state saturation. Analysis of the recurrent update suggests that persistent decay keeps weakly updated entries small, whereas uneven injections allow a few channels to accumulate extreme values. Motivated by this diagnosis, we propose \textbf{State Anomaly Neutralization (SANE)}, which applies adaptive $\tanh$ compression at chunk boundaries while preserving the intra-chunk parallel structure. Within a safe threshold range ($3 \le α\le 5$), SANE matches the baseline on 11 short-context reasoning benchmarks with no statistically significant degradation. After a 100M-token prefix, which exceeds the training length by over $24{,}000\times$, SANE retains functional reasoning ($33.46$--$35.56$) while the baseline encounters numerical overflow. In contrast, overly permissive thresholds ($α\ge 8$) remain numerically stable but lose reasoning capability entirely, showing that numerical stabilization alone does not guarantee functional reasoning and revealing a capacity--stability trade-off in state compression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。