MixFormer通过专家记忆机制提升长序列建模能力。
MixFormer: Linear Transformer with Mixture of Memory Experts
- 引入多专家记忆机制，分角色维护历史信息。
- 在文本与图像生成任务中显著优于现有线性Transformer。
- 适合需要高效处理超长序列的系统级应用。
状态空间模型（SSMs）作为线性Transformer的主流方向，旨在实现比标准Transformer更高的长序列建模效率。然而，现有SSMs存在输入适应性有限和记忆容量受限的问题，导致超长序列建模时信息丢失。为此，我们提出MixFormer，一种新型线性Transformer，集成混合记忆专家（MoE）机制。模型通过多个协同的记忆专家维护差异化的记忆状态，并采用新颖的时间感知线性注意力（TALA）机制，利用可学习的指数衰减函数和位置偏置动态更新记忆。该设计使模型能选择性强化重要历史信息，有效缓解记忆稀释问题，显著提升长程依赖建模能力。在长序列文本与图像生成任务上的实验表明，MixFormer不仅取得显著性能提升，还为下一代网络基础设施提供了更可持续的计算基础。
原文摘要 · Abstract (English)
State Space Models (SSMs), as a mainstream research direction of linear Transformers, aim to achieve higher efficiency than standard Transformers in long-context modeling. However, existing SSMs suffer from limited input adaptivity and constrained memory capacity, leading to information loss when modeling ultra-long sequences. To address these limitations, we propose MixFormer, a novel linear Transformer that integrates a Mixture-of-Memory-Experts (MoE) mechanism. Specifically, the model maintains differentiated memory states through multiple collaborating memory experts and employs a novel Time-Aware Linear Attention (TALA) mechanism, which leverages learnable exponential decay functions and positional biases to dynamically update memory. This design enables the model to selectively reinforce important historical information while effectively mitigating memory dilution, substantially improving long-range dependency modeling. Experiments on long-sequence text and image generation tasks demonstrate that MixFormer not only achieves significant performance gains but also provides a more sustainable computational backbone for the next generation of web infrastructure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。