提出PoST框架，让线性递归模型长程记忆更稳定高效。
Optimal Decay Spectra for Linear Recurrences
- 通过谱重参数化与位置自适应缩放，构建最优衰减谱。
- 长上下文下误差率降至exp(-cN/log t)，显著优于传统方法。
- 可无缝集成至多种模型，提升语言建模与长文本检索性能。
线性递归模型虽具备线性时间处理能力，但常因长程记忆表现不佳而受限。问题根源在于衰减谱：对于N个通道，随机初始化会使最小谱隙缩小至O(N⁻²)，导致误差呈次指数级exp(-Ω(N/log N))；线性间隔虽避免坍塌，却退化为exp(-O(N/√T))，实际为代数衰减。为此，我们提出位置自适应谱调制（PoST），一种架构无关的框架，包含两项机制：(1) 谱重参数化，结构上强制实现几何分布的对数衰减速率，理论证明其达到最小最大误差率O(exp(-cN/log T))；(2) 位置自适应缩放，唯一能消除静态谱尺度不匹配的机制——在位置t时仅约N log t / log T个通道有效，该机制将谱拉伸至实际依赖范围，使速率提升至O(exp(-cN/log t))。该缩放天然诱导分数阶不变性：脉冲响应变为尺度无关，通道在相对与绝对时间坐标间插值。PoST可无开销集成至任意对角线线性递归模型中。我们在Mamba-2、RWKV-7、Gated DeltaNet、Gated Linear Attention和RetNet上进行实例化，预训练规模达180M–440M，均展现一致的零样本语言建模提升，对Mamba-2在MQAR和NIAH任务上实现显著长上下文检索增益，并在其他架构上达到或超越原有性能。代码见：https://github.com/SiLifen/PoST。
原文摘要 · Abstract (English)
Linear recurrent models offer linear-time sequence processing but often suffer from suboptimal long-range memory. We trace this to the decay spectrum: for $N$ channels, random initialization collapses the minimum spectral gap to $O(N^{-2})$, yielding sub-exponential error $\exp(-Ω(N/\log N))$; linear spacing avoids collapse but degrades to $\exp(-O(N/\sqrt{T}))$, practically algebraic over long contexts. We introduce Position-Adaptive Spectral Tapering (PoST), an architecture-agnostic framework combining two mechanisms: (1) Spectral Reparameterization, which structurally enforces geometrically spaced log-decay rates, proven minimax optimal at rate $O(\exp(-cN/\log T))$; and (2) Position-Adaptive Scaling, the provably unique mechanism that eliminates the scale mismatch of static spectra (where only $N\log t/\log T$ of $N$ channels are effective at position $t$) by stretching the spectrum to the actual dependency range, sharpening the rate to $O(\exp(-cN/\log t))$. This scaling natively induces fractional invariance: the impulse response becomes scale-free, with channels interpolating between relative and absolute temporal coordinates. PoST integrates into any diagonal linear recurrence without overhead. We instantiate it across Mamba-2, RWKV-7, Gated DeltaNet, Gated Linear Attention, and RetNet. Pre-training at 180M-440M scales shows consistent zero-shot language modeling improvements, significant long-context retrieval gains for Mamba-2 (MQAR and NIAH), and competitive or improved performance across other architectures. Code: https://github.com/SiLifen/PoST.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。