提出新型注意力机制，实现精确的周期性长序列记忆。
Semidirect Fourier Delta Attention: Phase-Controlled Delta Memory with Constructive Chunk-WY Kernels
- 用傅里叶相位控制替代传统衰减，构建可调控的记忆块
- 数学证明其分块转移为精确仿射变换，内存增长可控
- 在小规模实验中展现周期记忆能力，优于基线模型
线性注意力用固定递归状态替代softmax注意力中随长度增长的键值缓存，但这种压缩限制了精确状态追踪与长程记忆。我们提出半直接傅里叶增量注意力（SFDA），作为Kimi Delta Attention的相位控制推广，将实数对角衰减替换为块旋转傅里叶控制：$ S_t=(I-β_t k_tk_t^*)Λ_tS_{t-1}+β_tk_tv_t^*, \quad Λ_t=\diag(α_t\odot e^{iθ_t}) $。主要成果是构造性地给出了乘积 $A_t=Λ_t-u_tr_t^*$ 的分块-WY分解，使得 $ A_t\cdots A_1=Γ_t-Y_tM_tW_t^* $，且秩增长被限定在固定块内。该结果带来精确仿射分块转移、形式稳定性与复杂度界，并实现了相位+低秩记忆的紧凑刻画。数值验证了代数正确性，并在简易状态追踪实验中显示，SFDA学习到周期性记忆，而相位禁用的基线模型表现接近随机。融合核与大规模语言模型对比留待未来工作。
原文摘要 · Abstract (English)
Linear attention replaces softmax attention's growing KV cache with a fixed recurrent state, but this compression limits exact state tracking and long-context memory. We introduce \emph{Semidirect Fourier Delta Attention} (SFDA), a phase-controlled generalization of Kimi Delta Attention that replaces real diagonal decay with block-rotational Fourier control: \[ S_t=(I-β_t k_tk_t^*)Λ_tS_{t-1}+β_tk_tv_t^*, \qquad Λ_t=\diag(α_t\odot e^{iθ_t}). \] Our main result is a constructive chunk-WY factorization for products \(A_t=Λ_t-u_tr_t^*\), giving \[ A_t\cdots A_1=Γ_t-Y_tM_tW_t^* \] with rank growth bounded inside fixed chunks. This yields an exact affine chunk transfer, formal stability and complexity bounds, and a compact characterization of phase-plus-low-rank memory. We verify the algebra numerically and show in toy state-tracking experiments that SFDA learns cyclic memory where the phase-disabled KDA baseline remains near chance. Fused kernels and large-scale language-model comparisons are left to future work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。