arXiv:2606.23957cs.LGcs.SY2026-06被引 1

用注意力免去的记忆块提升长期预测稳定性

Learning the Koopman Operator using Attention Free Transformers

论文配图:Learning the Koopman Operator using Attention Free Transformers
图 1 · 摘自论文原文
  • 引入无注意力的潜空间记忆模块,线性时间计算且仅增3万参数
  • 动态重编码机制实时检测潜空间漂移并纠正预测轨迹
  • 在1000步长时序预测中显著降低误差,适合高精度长期建模

使用自编码器学习Koopman算子可在隐空间实现线性预测,但长时间滚动预测常因脱离学习流形而产生相位与振幅误差,尤其在切换系统、连续谱或强瞬态系统中。本文提出两个互补组件:首先引入无注意力潜空间记忆(AFT)模块,通过短窗口历史隐状态聚合生成修正隐状态,每次更新前进行调整;相比多头注意力,AFT以线性时间运行,仅增加约3万参数(3d² + T²),却能有效捕捉局部时间上下文以抑制误差发散。其次提出动态重编码:采用轻量级在线变化点检测器(如EWMA、CUSUM及序列两样本检验),识别隐空间漂移后将预测投影回自编码器流形。在三个基准系统——杜芬振子、转录抑制环(Repressilator)、IRMA——上,模型持续优于基线Koopman自编码器和等容量多头注意力模型。与GRU及Transformer自编码器对比,无论从初始条件还是50步上下文出发,本方法在长时程预测中误差更低,推理延迟更小。报告了长达1000步的改进结果,并包含触发策略消融实验。最终获得一个快速、紧凑且长期保持流形一致性的预测器。

原文摘要 · Abstract (English)

Learning Koopman operators with autoencoders enables linear prediction in a latent space, but long-horizon rollouts often drift off the learned manifold, leading to phase and amplitude errors on systems with switching, continuous spectra, or strong transients. We introduce two complementary components that make Koopman predictors more robust. First, we add an attention-free latent memory (AFT) block that aggregates a short window of past latents to produce a corrected latent before each Koopman update. Unlike multi-head attention, AFT operates in linear time and adds only $\approx$30k parameters ($3d^2 + T^2$, fewer than matched multi-head attention), yet captures the local temporal context needed to suppress error divergence. Second, we propose dynamic re-encoding: lightweight, online change-point triggers (EWMA, CUSUM, and sequential two-sample tests) that detect latent drift and project predictions back onto the autoencoder manifold. Across three benchmark systems -- Duffing oscillator, Repressilator, IRMA -- our model consistently reduces error accumulation compared to a Koopman autoencoder and matched-capacity multi-head attention. We also compare against GRU and Transformer autoencoders, evaluated both from initial conditions and with a 50-step context, and find that Koopman+AFT (with optional re-encoding) attains markedly lower long-horizon error while maintaining lower inference latency. We report improvements over horizons up to 1000 steps, together with ablations over trigger policies. The result is a fast, compact predictor that stays on the learned manifold over long horizons.

Koopman算子长期预测记忆模块动态重编码

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。