让Transformer学会按幂律记忆远距离信息，更符合语言真实依赖结构。
VORT: Adaptive Power-Law Memory for NLP Transformers
- 用可学习的分数阶参数控制记忆衰减，实现非马尔可夫的长程依赖建模。
- 在两个合成任务中均优于传统方法，尤其在均匀延迟的实体复制任务上表现显著。
- 适合研究长文本建模、自适应记忆机制的NLP研究人员。
标准Transformer对远距离词元的影响呈近指数衰减，与自然语言中长程依赖的幂律结构相悖。我们提出变量阶保留Transformer（VORT），为每个输入词元分配一个可学习的分数阶α_i∈[δ,1]，通过Grünwald–Letnikov幂律保留核进行建模。由于分数加权和是非马尔可夫的，我们通过拉普拉斯型积分表示下的高斯-勒让德求积法，将其近似为多项式指数和（SOE），每步复杂度为O(Sd_v)，其中S=O(log(T/ε))项即可在[1,T]区间内达到ε-一致精度。检索通过线性注意力累加器实现，每步开销为O(KSd_ϕd_v)。本文建立了四项成果：(i) SOE逼近定理，基于变量变换后的被积函数解析性，具有几何收敛率；(ii) 在[δ,1]区间内的量化误差界，正确处理α接近0的情况；(iii) 直接的L²能量论证表明，当α>1/2时，任何固定最小衰减速率Λ>0的混合形式在L²([1,T])上的误差至少为N_α(T)−C(Λ)→∞，且Λ依赖关系明确；(iv) 在Polyak–Łojasiewicz条件下，梯度塑性规则呈现线性收敛。两个合成实验验证了架构优势：一个服从齐普夫分布的检索基准，以及一个具有均匀延迟分布的实体标签复制任务，后者排除了先验匹配作为幂律核优势的解释。
原文摘要 · Abstract (English)
Standard Transformers impose near-exponential decay on the influence of distant tokens, conflicting with the power-law structure of long-range dependencies in natural language. We introduce the \emph{Variable-Order Retention Transformer} (\VORT{}), a memory architecture in which each ingested token is assigned a learnable fractional order α_i\in[δ,1] that governs a Grünwald--Letnikov power-law retention kernel. Because the fractional weighted sum is non-Markovian, we approximate it through a sum-of-exponentials (SOE) decomposition computed by Gauss--Laguerre quadrature on a Laplace-type integral representation of the kernel weights. Each exponential component admits a one-step Markovian recurrence at O(Sd_v) per step, where S=O(\log(T/\varepsilon)) terms suffice for \varepsilon-uniform accuracy on horizon [1,T]. Retrieval is keyed and associative via a linear-attention accumulator with an exact O(KSd_ϕd_v) -per-step recurrence. Four results are established: (i) an SOE approximation theorem with geometric convergence rate from the analyticity of the integrand after a log-change of variables; (ii) a quantisation bound valid on [δ,1] with correct analysis near α=0; (iii) a direct L^2 energy argument (Proposition) showing that for α>1/2 any mixture with fixed minimum decay rate Λ>0 incurs L^2([1,T]) error at least N_α(T)-C(Λ)\to\infty, with the Λ-dependence made explicit; and (iv) linear convergence of a gradient plasticity rule under the Polyak--Łojasiewicz condition. Two synthetic experiments confirm the architectural advantage: a Zipf-distributed retrieval benchmark and an entity label-copy task with uniform lag distribution, the latter ruling out prior-matching as an explanation for the power-law kernel's advantage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。