arXiv:2605.23603cs.LGcond-mat.dis-nn2026-05被引 3

用磁滞模型重构注意力,让序列记忆更高效。

Preisach Attention: A Hysteretic Model of Sequential Memory

  • 用二元继电器替代softmax,基于局部极值构建记忆栈。
  • 单层PAL-Transformer可模拟下推自动机,深度仅需O(1)。
  • 适合长时记忆任务,对位置不敏感,推理成本更低。

我们提出预斯哈奇注意力层(Preisach Attention Layer, PAL),基于数学物理中的经典磁滞算子构建序列建模架构。PAL 用参数化激活/去激活阈值的二元继电器替代 softmax 注意力,以局部极值堆栈作为内部状态。单层 PAL-Transformer 在任意精度算术下具备图灵完备性,可通过模拟双栈下推自动机实现,仅需 O(1) 深度,远低于标准硬注意力变换器所需的 O(log n)。其次,我们证明了 PAL 与 Transformer 的函数类不可比较:PAL 可在 O(1) 层内计算历史极值统计量,而 Transformer 需 O(log n) 层;但后者支持随机访问检索,PAL 无法无辅助状态实现。该分离特性源于速率无关性——PAL 仅响应局部极值序列,不依赖绝对位置或时间间隔。第三,我们证明极值堆栈是所有速率无关泛函的最小充分统计量,形式上对应经典磁滞理论中的‘擦拭’性质。因此,PAL 是长时情景记忆且位置依赖弱的任务的高效架构,总推理成本为 O(n log n),远低于标准注意力的 O(n²)。

原文摘要 · Abstract (English)

We introduce the Preisach Attention Layer (PAL), a novel sequence modelling architecture grounded in the classical Preisach hysteresis operator from mathematical physics. PAL replaces the softmax attention mechanism with a binary relay operator parameterised by learned activation and deactivation thresholds, maintaining a stack of local extrema as its internal state. A single-layer PAL-Transformer with O(1) depth is Turing-complete under arbitrary precision arithmetic, achievable through simulation of a two-stack pushdown automaton -- in contrast to the O(log n) depth required by standard hard-attention transformers. Second, we prove that the function classes computable by PAL and by the transformer are incomparable: PAL computes historical range statistics in O(1) layers that require O(log n) layers for transformers, while transformers support random-access retrieval that PAL cannot perform without auxiliary state. The separating property is rate-independence -- PAL responds only to the sequence of local extrema, not to absolute token positions or temporal spacing. Third, we show that the extremum stack constitutes a minimal sufficient statistic of the input history for all rate-independent functionals, providing a formal analogue of the wiping property in classical hysteresis theory. PAL is thus an efficient architecture for tasks with long episodic memory and weak positional dependence, with O(n log n) total inference cost versus O(n^2) for standard attention.

注意力机制序列建模磁滞模型记忆效率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。