FDM模型实现恒定内存解码，突破传统注意力机制瓶颈。
Breaking the KV Cache Bottleneck: Fan Duality Model Achieves O(1) Decode Memory with Superior Associative Recall
- 分离波与粒子组件，用固定大小状态压缩长序列
- 解码内存恒定867MB，较Transformer减少4.9倍
- 适合需要高效推理的长文本生成场景
我们提出FDM（Fan Duality Model），一种线性序列架构，解决了序列建模中内存效率与关联召回之间的根本矛盾。FDM将序列处理分为两个部分：波组件（通过保相Givens旋转进行递归扫描），将长程模式压缩为固定大小的复数隐藏状态；粒子组件（局部-全局缓存），通过学习的关联寻址检索特定词元，使用W+K=272个独立于序列长度N的槽位。这实现了严格的O(1)解码内存：在所有提示长度128-8,192词元下保持867MB，而Transformer为853-4,247MB（N=8,192时降低4.9倍）。此外，发现联合训练波与粒子组件会导致收敛不佳，提出冻结扫描的两阶段训练策略，使WikiText-103上PPL=64.9（44K步），较全微调提升7.5倍（原为487）。在多查询关联召回（MQAR）任务中，FDM准确率达0.966，超越Transformer的0.606（提升59.5%），纯扫描无缓存仅0.011，证明粒子组件必要性。最后引入全息参考束解码，将隐藏状态h_t视为编码整个时间历史的全息板，以当前输入x_t作为参考束调制h_t，使用4头正交参考束仅增加130万参数，使PPL下降2.13点至62.79，验证了全息解释的合理性。代码与预训练权重见：https://github.com/YasongFan/FDM
原文摘要 · Abstract (English)
We present FDM (Fan Duality Model), a linear sequence architecture that resolves the fundamental tension between memory efficiency and associative recall in sequence modeling. FDM separates sequence processing into two components: a wave component (recurrent scan via phase-preserving Givens rotations) that compresses long-range patterns into a fixed-size complex hidden state, and a particle component (local-global cache) that retrieves specific tokens via learned associative addressing with W+K=272 slots independent of sequence length N. This yields strictly O(1) decode memory: 867 MB fixed across all prompt lengths 128-8,192 tokens, versus Transformer's 853-4,247 MB (4.9x reduction at N=8,192). Beyond the architecture, we discover that jointly training the wave and particle components leads to suboptimal convergence. We propose Freeze-Scan, a two-phase training strategy that freezes the recurrent scan and optimizes the cache jointly with embeddings, achieving PPL=64.9 on WikiText-103 in 44K steps -- a 7.5x improvement over full fine-tuning (PPL=487). On Multi-Query Associative Recall (MQAR), FDM achieves 0.966 accuracy, surpassing Transformer (0.606) by 59.5%, while pure scan without cache scores only 0.011, confirming the necessity of the particle component. Finally, we introduce Holographic Reference Beam Decoding, interpreting the complex hidden state h_t as a holographic plate encoding the entire temporal history. Using the current input x_t as a reference beam to modulate h_t reduces PPL by up to 2.13 points (PPL=62.79) with a 4-head orthogonal reference beam using only 1.3M additional parameters, providing empirical support for the holographic interpretation. Code and pretrained weights: https://github.com/YasongFan/FDM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。