提出可移动坐标系下的记忆投影机制,提升序列模型对历史信息的敏感性。
SHiPPO: Recurrent Memory with Transported Polynomial Projections

- 将传统HiPPO的记忆系数从固定坐标转为随输入动态移动的坐标系
- 在有限域上实现记忆重写与查询的联合测试,验证了运输路径的有效性
- 适合研究记忆机制的模型设计者,尤其关注历史依赖关系建模
HiPPO通过在线多项式投影的系数赋予循环状态记忆语义,但限定在固定通道坐标。现代选择性SSM则依赖于与标记相关的控制和通道交互。我们提出SHiPPO(Sylvester HiPPO),一种将HiPPO系数记忆提升至移动通道框架的传输投影-记忆先验。对于任意固定或实际的右向传输路径,SHiPPO同步传输近似族与通道度量;在该路径条件下,状态表现为绑定移动框架中的普通HiPPO,并遵循Sylvester系数动力学,保留左向在线记忆算子的同时引入右向作用的传输。针对选择性SSM执行,我们推导出具有控制器兼容右作用、指数调整更新、精确块仿射扫描及递归解码的受限群局部实现。同时给出同时可约性判据,识别出右向传输退化为静态混合加独立标量或分块银行的情形。受控诊断显示,更大的当前标记写入秩虽能降低常规预测误差,却无法恢复已写记忆中顺序敏感变化;而传输记忆变体可恢复这一信号,该信号在移除传输路径后消失。一个包含交错绑定、操作与查询的有限域关联回忆诊断提供互补的自回归证据,同时保持首选右作用实现开放。综合来看,这些结果支持SHiPPO作为机制上合理的传输记忆先验,重点聚焦于记忆机制而非广泛的序列建模优势。
原文摘要 · Abstract (English)
HiPPO gives recurrent states memory semantics as coefficients of online polynomial projections, but in fixed channel coordinates. Modern selective SSMs, by contrast, rely on token-dependent control and channel interaction. We introduce SHiPPO (Sylvester HiPPO), a transported projection-memory prior that lifts HiPPO coefficient memories into a moving channel frame. For any fixed or realized right-transport path, SHiPPO transports the approximation family and channel metric together; conditional on that path, the state is ordinary HiPPO in a tied moving frame and follows Sylvester coefficient dynamics, preserving the left online-memory operator while adding right-action transport. For selective-SSM execution, we derive a restricted group-local realization with controller-compatible right actions, exponential-adjusted updates, exact block-affine scan, and recurrent decoding. We also give a simultaneous-reducibility criterion identifying when right transports collapse to static mixing plus independent scalar or blockwise banks. Controlled diagnostics show that larger current-token write rank improves ordinary prediction error but cannot recover order-sensitive changes to already-written memory; transported-memory variants recover this signal, which disappears when the transport pathway is removed. A finite-field associative-recall diagnostic with interleaved bindings, operations, and queries provides complementary autoregressive evidence while leaving the preferred right-action realization open. Taken together, these results support SHiPPO as a mechanistically grounded transported-memory prior, with evidence focused on memory mechanisms rather than broad sequence-modeling dominance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。